Papers with natural language processing tasks
Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing models with unstructured pruning often yield irregular sparsity patterns that necessitate specialized hardware or software support. |
| Approach: | They propose a structured pruning framework that eliminates entire architectural components and maintains compatibility with standard hardware accelerators. |
| Outcome: | The proposed model pruning framework achieves significant compression with minimal performance degradation on multiple models across diverse downstream tasks. |
Investigation on Data Adaptation Techniques for Neural Named Entity Recognition (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition use only a limited number of samples . data augmentation and selftraining are popular methods to generate additional synthetic data . |
| Approach: | They investigate the impact of data augmentation and data augmented on named entity recognition tasks. |
| Outcome: | The proposed methods improve the performance of three named entity recognition tasks. |
Are LLMs Good Annotators for Discourse-level Event Relation Extraction? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks, but their effectiveness over discourse-level event relation extraction tasks remains unexplored. |
| Approach: | They evaluate LLMs' ability to address discourse-level event relation extraction tasks using an open-source model and a commercial model. |
| Outcome: | The proposed model performs poorly on discourse-level event relation extraction tasks. |
Computational Linguistics for Brain Encoding and Decoding: Principles, Practices and Beyond (2024.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will explore the potential of computational linguistics to help understand brain language processing. |
| Approach: | This tutorial will explore the principles and practices of using computational linguistics methods for brain encoding and decoding. |
| Outcome: | This tutorial will explore the principles and practices of using computational linguistics methods for brain encoding and decoding. |
A New Approach to Animacy Detection (C18-1)
Copied to clipboard
| Challenge: | Animacy is a property for a referent to be an agent, and prior work has classified words as either animate or inanimate. |
| Approach: | They propose a method that uses supervised machine learning and hand-built rules to classify the animacy of co-reference chains. |
| Outcome: | The proposed method achieves state-of-the-art performance on a 142-text dataset . it leverages word embeddings over referring expressions, parts of speech, and grammatical and semantic roles . |
Comparing the Intrinsic Performance of Clinical Concept Embeddings by Their Field of Medicine (D19-62)
Copied to clipboard
| Challenge: | Existing work has trained medical embeddings to rep-resent medical concepts using specific medical data. |
| Approach: | They use intrinsic methods to evaluate pre-trained word embeddings from the various fields of medicine as defined by their ICD-9 systems. |
| Outcome: | The results show that the embeddings perform better in one field of medicine than in other fields. |
Robust to Noise Models in Natural Language Processing Tasks (P19-2)
Copied to clipboard
| Challenge: | Existing spelling correction systems are far from perfect for noise-sensitive texts . a new way to handle noise is to make models robust to noise. |
| Approach: | They propose a robust to noise word embeddings model which outperforms existing models in different tasks. |
| Outcome: | The proposed model outperforms existing models in three downstream tasks and shows improvements in noise robustness over existing models. |
Toward Building a Language Model for Understanding Temporal Commonsense (2022.aacl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT are still poor in temporal reasoning . commonsense reasoning is crucial for natural language processing (NLP) |
| Approach: | They propose to use multi-step fine-tuning and masked language modeling to predict mangled temporal indicators that are crucial for commonsense reasoning. |
| Outcome: | The proposed model improves performance on multiple time-related tasks. |
Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have used pre-trained language models for domain adaptation or transfer learning to improve natural language processing performance. |
| Approach: | They propose to fine-tune word sense disambiguation on sequence-pair ranking task and to use existing WordNet examples to augment the model. |
| Outcome: | The proposed model achieves state-of-the-art on the English all-words benchmark datasets. |
ET: A Workstation for Querying, Editing and Evaluating Annotated Corpora (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Using annotated corpora is costly for humans alone and requires a large amount of time and effort to manipulate. |
| Approach: | They propose to use annotated corpora as a tool for linguistic research and natural language processing. |
| Outcome: | The proposed work is based on two integrated environments – Interrogatório and Julgamento . the open-source environment is used in several linguistic and NLP-related studies . |
FITAnnotator: A Flexible and Intelligent Text Annotation System (2021.naacl-demos)
Copied to clipboard
| Challenge: | In this paper, we introduce FITAnnotator, a generic web-based tool for efficient text annotation. |
| Approach: | They propose a generic web-based tool for efficient text annotation. |
| Outcome: | The proposed tool is based on a fully modular architecture and provides three kinds of interfaces to annotate instances, evaluate annotation quality and manage the annotation task for annotators, reviewers and managers. |
Multi-Task, Multi-Channel, Multi-Input Learning for Mental Illness Detection using Social Media Text (D19-62)
Copied to clipboard
| Challenge: | Existing methods for mental illness detection have limited data available for training . lack of sufficient annotated data and inability to extract explanations on the derived outcome have restricted researchers to use traditional methods. |
| Approach: | They propose to use emotional patterns identified by clinical practitioners to enhance the prediction capabilities of a mental illness detection model built using a deep neural network architecture. |
| Outcome: | The proposed method achieves a task-specific AUC higher than 0.90 . it compares multi-task learning with multi-channel convolutional neural network and multiple inputs to methods such as multi-class classification . |
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2020.acl-main)
Copied to clipboard
| Challenge: | Existing pre-training models for dialogue generation have been proven effective for a wide range of tasks. |
| Approach: | They propose a dialogue generation pre-training framework that leverages bi-directional context and uni-directional characteristic of language generation. |
| Outcome: | The proposed framework is superior to existing models on three publicly available datasets. |
Meta-Learning Adaptive Knowledge Distillation for Efficient Biomedical Natural Language Processing (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods have been proposed to reduce the size of large models for biomedical natural language processing tasks. |
| Approach: | They propose a meta-learning approach which adaptively learns parameters that enable optimal rate of knowledge exchange between teacher and student models from the distillation data during knowledge distillation. |
| Outcome: | The proposed method improves the performance of knowledge distillation methods on two biomedical natural language processing tasks. |
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Labelled data is the foundation of most natural language processing tasks, but there are valid beliefs about what the correct data labels should be. |
| Approach: | They propose two contrasting paradigms for data annotation that encourage annotator subjectivity . they propose a descriptive paradigm that allows for the surveying and modelling of different beliefs . |
| Outcome: | The proposed paradigms encourage annotator subjectivity, while the prescriptive paradigm discourages it. |
An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media (D19-55)
Copied to clipboard
| Challenge: | Existing natural language processing tools are focused on standard texts, but performance drops when used on a different domain. |
| Approach: | They analyze the effect of manual and automatic lexical normalization for dependency parsing . they conclude that automatic normalization scores close to manually annotated normalization . |
| Outcome: | The proposed approach improves performance on social media data for many tasks . it is unclear which replacements have the most impact and what weaknesses exist in the system . |
COMMA: Modeling Relationship among Motivations, Emotions and Actions in Language-based Human Activities (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for modeling motivations, emotions and actions in language-based human activities have been limited. |
| Approach: | They propose to model motivations, emotions and actions in language-based human activities using a dataset called Story Commonsense. |
| Outcome: | The proposed model can better reveal the essential relationship between motivations, emotions and actions than existing methods. |
PromptLab: A Collaborative Platform for Prompt Engineering and Dataset Curation (2026.eacl-demo)
Copied to clipboard
Maged S. Al-shaibani, Zaid Alyafeai, Dania Refai, Nawaf Alomari, Ahmed Ashraf, Mais Alheraki, Mustafa Alturki, Hamzah Luqman, Irfan Ahmad
| Challenge: | PromptLab is a web-based prompt engineering platform for collaborative prompt development across diverse natural language processing tasks and datasets. |
| Approach: | They propose to integrate prompt generation via OpenRouter and provide real-time validation with multiple Large Language Models. |
| Outcome: | The platform addresses primary challenges in prompt development, including template creation, collaborative review, and quality assurance through a comprehensive workflow that supports both individual researchers and team-based projects. |
Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP (2021.eacl-demos)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has become a standard repertoire in natural language processing (NLP) it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters. |
| Approach: | They propose a toolkit for fine-tuning contextualized embeddings in multi-task settings. |
| Outcome: | The proposed toolkit supports a variety of natural language processing tasks . it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters. |
A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation (2022.findings-aacl)
Copied to clipboard
| Challenge: | Current architectures are unable to integrate knowledge base schema and handle questions unseen during training rendering them unusable outside the scope of topics covered in the training set. |
| Approach: | They propose to integrate a copy mechanism for neural SPARQL query generation by adding a knowledge base layer and a dynamic knowledge base vocabulary to two Seq2Seq architectures. |
| Outcome: | The proposed model outperforms existing models on state-of-the-art datasets and shows a significant increase in performance. |
Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing theoretical frameworks for large language models (LLMs) do not explain how pretraining leads to in-context learning. |
| Approach: | They propose a theoretical framework that allows LLMs to generalize to unseen instructions and perform in-context learning even when verbalizers are irrelevant to the task. |
| Outcome: | The proposed framework can be used to analyze LLMs' ability to perform in-context learning . it can be applied to linguistic, psychology, and philosophy tasks . |
ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing frameworks for large language model embeddings have limited support for only a limited range of architectures and fine-tuning strategies. |
| Approach: | They propose a framework that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
| Outcome: | The proposed framework enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
A Representation Learning Approach to Animal Biodiversity Conservation (2020.coling-main)
Copied to clipboard
| Challenge: | Using vector embeddings, we can predict the conservation status of animal species using a machine learning model. |
| Approach: | They propose a method for predicting the conservation status of animal species using custom generated name embeddings from Wikipedia text and animal taxonomy data. |
| Outcome: | The proposed model predicts the conservation status of data deficient species and steers more focus towards them for protection. |
Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have brought significant improvements to many natural language processing tasks. |
| Approach: | They propose a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network. |
| Outcome: | The proposed method improves on several NLI datasets using a strong baseline model (RoBERTa-large) with MNLI pretraining. |
LogRules: Enhancing Log Analysis Capability of Large Language Models through Rules (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) exhibit hallucinations when analyzing logs due to the implicit knowledge and rules in logs that LLMs cannot capture. |
| Approach: | They propose a lightweight log analysis framework that generates and utilizes rules through LLMs. |
| Outcome: | The proposed framework outperforms LLM-based methods in log parsing and anomaly detection tasks and achieves better performance compared to case-based approaches. |
Can Large Language Models Accurately Generate Answer Keys for Health-related Questions? (2025.acl-short)
Copied to clipboard
| Challenge: | Evaluating the factuality of LLM generated answers is challenging for many tasks, including question answering. |
| Approach: | They propose to use information nuggets to evaluate the factuality of LLM generated answers . they find providing an example and extracting nuggots from an answer is the best approach . |
| Outcome: | The proposed model performs best when compared to human nugget generation. |
Dynamic Task-Oriented Dialogue: A Comparative Study of Llama-2 and Bert in Slot Value Generation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in instruction-based language models have demonstrated exceptional performance across various natural language processing tasks. |
| Approach: | They propose to use BERT and Llama-2 to generate dynamic task-oriented dialogues using a multi-dimensional dataset. |
| Outcome: | The proposed models generate predictions for masked slot values within text and are reproducible in open-source environments. |
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation remains challenging due to pretraining on predominantly English-centric datasets. |
| Approach: | They propose a method that combines reward scores with model confidence to improve model selection for fine-tuning. |
| Outcome: | The proposed method outperforms existing methods in translation accuracy and data efficiency. |
Exploring Sentence Community for Document-Level Event Extraction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to document-level event extraction neglect the complex logic structures in long texts. |
| Approach: | They propose a framework that exploits the relationship between sentences to extract multiple events by sentence community detection using graph attention networks. |
| Outcome: | The proposed framework achieves competitive results over state-of-the-art methods on the large-scale document-level event extraction dataset. |
Grammar-Constrained Decoding Makes Large Language Models Better Logical Parsers (2025.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown capabilities in various natural language processing tasks, yet struggle with logical reasoning. |
| Approach: | They propose to combine Large Language Models with symbolic reasoners to improve syntactic correctness and semantic accuracy in logical parsing tasks. |
| Outcome: | The proposed approach improves syntactic correctness and semantic accuracy in logical parsing tasks. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | masked language models have been used for natural language processing tasks but few studies have adopted it in the sequence-to-sequence models. |
| Approach: | They propose to combine encoder and decoder to train a masked sequence-to-sequence model . they propose to train the encoder more rigorously by masking the encoded input . |
| Outcome: | The proposed model achieves 27.69/32.24 BLEU scores on English-German/German-English tasks with 5+ times speed up compared with an autoregressive model. |
Exploiting Language Model Prompts Using Similarity Measures: A Case Study on the Word-in-Context Task (2022.acl-short)
Copied to clipboard
| Challenge: | Existing few-shot approaches fail on the semantic distinction task of the Word-in-Context dataset. |
| Approach: | They propose a prompt-based approach which boosts few-shot performance to the level of fully supervised methods by using similarity metrics. |
| Outcome: | The proposed technique boosts few-shot performance to the level of fully supervised methods. |
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based pre-training models like BERT are computationally expensive and limited to resource-constrained devices. |
| Approach: | They propose a method which ternarizes the weights in a fine-tuned BERT model. |
| Outcome: | The proposed method outperforms the other methods on the GLUE and SQUAD benchmarks while being 14.9x smaller. |
eMLM: A New Pre-training Objective for Emotion Related Tasks (2021.acl-short)
Copied to clipboard
| Challenge: | Emotion Masked Language Modelling improves the performance of a pretraining language model for emotion detection and sentiment analysis tasks. |
| Approach: | They propose a BERT-based version of Masked Language Modelling that induces emotion into the model. |
| Outcome: | The proposed model improves on emotion detection and sentiment analysis tasks by 1.2% F-1 . the proposed model also shows increased robustness in the test. |
Where do LLMs Encode the Knowledge to Assess the Ambiguity? (2025.coling-industry)
Copied to clipboard
| Challenge: | False sizing of large language models can generate unreliable responses . |
| Approach: | They propose a method to train large language models without ambiguity labels . |
| Outcome: | The proposed method detects ambiguous input prompts better than representations from the final layer. |
What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Deep learning has demonstrated performance advantages in a wide range of natural language processing tasks. |
| Approach: | They propose to deepen the decoder layer in a Transformer model to reduce the difficulty of deep learning. |
| Outcome: | The proposed method can deepen the model on both the encoder and decoder at the same time, resulting in a deeper model and improved performance. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation? (N18-1)
Copied to clipboard
| Challenge: | Distributed representations of words learned from text have proved to be successful in various natural language processing tasks. |
| Approach: | They propose to embed a distributional thesaurus network into dense word vectors and compare them to state-of-the-art word representations. |
| Outcome: | The proposed representations improve performance against state-of-the-art word representations even without handcrafted lexical resources. |
Building Location Embeddings from Physical Trajectories and Textual Representations (2020.aacl-main)
Copied to clipboard
| Challenge: | Using a dataset consisting of the location trajectories of 729 students over a seven month period, we investigate whether embeddings can represent aspects such as location presence or location functionality. |
| Approach: | They propose to use location embeddings to generate embeddables of sequences of locations a student has visited to identify surface properties captured in the representations. |
| Outcome: | The proposed models can be used to predict depression levels and area of study, and can be applied to complex tasks such as predicting area of studies and depression levels. |
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way (2023.emnlp-demo)
Copied to clipboard
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, Xipeng Qiu
| Challenge: | Large language models (LLMs) are increasingly pivotal in a wide range of tasks . however, the resources required for training these models necessitate efficient solutions . |
| Approach: | They propose a library that facilitates collaborative training of large language models . they use 3D parallelism, parameter-efficient fine-tuning methods and optimizers . |
| Outcome: | The proposed library has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. |
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to the emergence of large language models capable ofvarious natural language processing tasks. |
| Approach: | They propose a multi-instructional training approach that integrates a large language model with a speech encoder to harness the capabilities of LLMs for speech recognition and beyond. |
| Outcome: | The proposed model can be trained and aligned with a multilingual LLM on 1900 hours of transcribed data from 139 languages. |
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence. |
| Approach: | They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes. |
| Outcome: | The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models. |
DisComp: A Two-Stage Prompt Optimization Framework Combining Task-Agnostic and Task-Aware Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Extended prompts can lead to substantial computational overhead and increased hardware demands, limiting the scalability and efficiency of large language models. |
| Approach: | They propose a two-stage prompt compression framework that combines task-agnostic and task-based strategies to efficiently compress prompt length without compromising performance. |
| Outcome: | The proposed framework outperforms task-agnostic and task-specific compression methods on three benchmark datasets and is up to 6.56 faster at inference compared to the best token-level compression method. |
Neural Latent Relational Analysis to Capture Lexical Semantic Relations in a Vector Space (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to capture semantic relations of words in vector space are lacking information on lexico-syntactic patterns that connect word pairs in a corpus. |
| Approach: | They propose a pattern-based approach that exploits lexico-syntactic patterns as word pairs . they propose NLRA to generalize co-occurrences of word pairs and lexicon-sensitized embeddings of the word pairs that do not co-occur. |
| Outcome: | The proposed model outperforms existing models on measuring relational similarity . it can generalize word pairs and lexico-syntactic patterns and obtain embeddings of word pairs that do not co-occur . |
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022.acl-short)
Copied to clipboard
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, Aram Galstyan
| Challenge: | Recent natural language processing systems use large language models as the backbone . however, societal biases are encoded in these models and transferred to downstream applications . |
| Approach: | They propose to use two categories to measure fairness in natural language processing tasks . they find intrinsic and extrinsic metrics do not correlate in their original setting . |
| Outcome: | The proposed metrics do not correlate in their original setting, the authors show . they find that they are not accurate when correcting for metric misalignments and noise . |
A Corpus to Learn Refer-to-as Relations for Nominals (L18-1)
Copied to clipboard
| Challenge: | Existing work on how to learn refer-to-as relations from large unlabeled corpora lacks coreferential information. |
| Approach: | They propose to use Wikipedia to generate coreferential neural embeddings for nominals . they use coreference resolution as a proxy to evaluate the neural embeds for noun phrases . |
| Outcome: | The proposed dataset can be leveraged to construct representations for coreferential nominals from Wikipedia. |
In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that in-context examples for machine translation are beneficial for high-resource languages. |
| Approach: | They propose to use in-context examples for machine translation (MT) they argue that similarity-based selection can improve MT . |
| Outcome: | The proposed approach improves machine translation (MT) and low-resource languages. |
SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (N19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are currently the dominant approach in natural language processing tasks, but require massive parallel corpora. |
| Approach: | They propose a sequence-to-sequence-tosequnce autoencoder with words as latent variables . they apply the model to unsupervised abstractive sentence compression . |
| Outcome: | The proposed model achieves promising results in unsupervised sentence compression on benchmark datasets. |
SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition (P19-1)
Copied to clipboard
| Challenge: | Selectional Preference (SP) is a common phenomenon in human language and has been shown to be useful in many natural language processing tasks. |
| Approach: | They propose a large-scale evaluation set that provides human ratings for the plausibility of 10,000 SP pairs over five SP relations, covering 2,500 most frequent verbs, nouns, and adjectives in American English. |
| Outcome: | The proposed evaluation sets provide human ratings for plausibility of 10,000 SP pairs over five SP relations covering 2,500 most frequent verbs, nouns, and adjectives in American English. |
Explaining Word Embeddings via Disentangled Representation (2020.aacl-main)
Copied to clipboard
| Challenge: | Disentangled representations are known to represent interpretable factors in separated dimensions. |
| Approach: | They propose to transform dense word vectors into disentangled embeddings with improved interpretability by encoding polysemous semantics separately. |
| Outcome: | The proposed model can be encoded into multiple sub-embeddings or sub-areas and generates more efficient and effective features for natural language processing. |
Exploring Cross-sentence Contexts for Named Entity Recognition with BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is often addressed as a sequence classification task with each input consisting of one sentence of text. |
| Approach: | They propose a method to combine different predictions from multiple sentences in input samples to increase NER performance. |
| Outcome: | The proposed method improves on the state-of-the-art NER results on English, Dutch, and Finnish and achieves the best reported BERT-based results on German. |
A Self-supervised Joint Training Framework for Document Reranking (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models have been successfully applied to a wide range of tasks . however, the pretraining tasks were based on the context of documents . |
| Approach: | They propose a self-supervised joint training framework with a method called Masked Query Prediction to establish semantic relations between given queries and positive documents. |
| Outcome: | The proposed framework outperforms existing models on document reranking tasks without further pre-training . it uses a self-supervised method to establish semantic relations between given queries and positive documents. |
Persian Ezafe Recognition Using Transformers and Its Role in Part-Of-Speech Tagging (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Ezafe is a grammatical particle in some Iranian languages that links two words together but is almost always not indicated in Persian script. |
| Approach: | They propose to use Persian ezafe to improve part-of-speech tagging by using transformer-based methods to achieve state-of the-art results. |
| Outcome: | The proposed methods achieve state-of-the-art in the task of ezafe recognition and show that they are not useful to transformer-based methods. |
Addressing Noise in Multidialectal Word Embeddings (P18-2)
Copied to clipboard
| Challenge: | Dialectal Arabic (DA) is problematically noisy and lacks a large corpus of non-noisy words. |
| Approach: | They propose to use word embedding tools to maximize the informative content leveraged in each training sentence and analyze methods for representing disparate dialects in one embeddable space. |
| Outcome: | The proposed methods improve performance on low and high frequency words while preserving accuracy on low frequency forms. |
Selecting Key Views for Zero-Shot Entity Linking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Entity linking is a task of assigning ambiguous mentions in textual input to entities in knowledge bases. |
| Approach: | They propose a framework to align mentions in text to entities in knowledge bases . they use unsupervised clustering to select key views from descriptions . |
| Outcome: | The proposed framework achieves state-of-the-art on the zero-shot entity linking dataset. |
Prompt Perturbation Consistency Learning for Robust Language Models (2024.findings-eacl)
Copied to clipboard
Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan
| Challenge: | Large language models have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization. |
| Approach: | They propose a method to reduce the performance drop of large language models by regularizing the divergence between losses from clean and perturbed samples. |
| Outcome: | The proposed approach recovers on average 59% and 69% of the performance drop for IC and SF tasks while using ten times fewer augmented data samples. |
Enhancing Neural Models with Vulnerability via Adversarial Attack (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on adversarial attack to improve performance of NLSM tasks has not been done. |
| Approach: | They propose a general two-stage training framework to enhance neural models with Vulnerability via adversarial attack. |
| Outcome: | The proposed framework improves neural models with Vulnerability via adversarial attack on NLSM datasets. |
Extracted BERT Model Leaks More Information than You Think! (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vulnerable to model extraction attacks . model extraction can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
| Approach: | They propose to launch an attribute-inference attack against an extracted BERT model to prevent privacy leakage. |
| Outcome: | The proposed attack can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
Convolutional Neural Networks with Recurrent Neural Filters (D18-1)
Copied to clipboard
| Challenge: | Convolutional neural networks (CNNs) use recurrent neural networks as convolution filters to capture language compositionality and long-term dependencies. |
| Approach: | They propose to use recurrent neural networks (RNNs) as convolution filters to capture language compositionality and long-term dependencies. |
| Outcome: | The proposed convolutional neural networks achieve state-of-the-art on two sentences and the Stanford Sentiment Treebank. |
Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing word embedding methods for natural language processing are limited in their ability to produce dense word embeds. |
| Approach: | They propose a word embedding SentiVec which is infused with sentiment information from a lexical resource and outperforms baselines on subjectivity-sensitive tasks. |
| Outcome: | The proposed word embedding SentiVec outperforms baselines on subjectivity-sensitive tasks. |
Word Embedding and WordNet Based Metaphor Identification and Interpretation (P18-1)
Copied to clipboard
| Challenge: | Existing models cannot identify exact metaphorical words within a sentence . current models do not rely on hand-crafted knowledge for training . |
| Approach: | They propose an unsupervised learning method that identifies and interprets metaphors at word-level without preprocessing. |
| Outcome: | The proposed method outperforms baseline models in two translation systems for English to Chinese showing that it paraphrases metaphors into their literal counterparts. |
BERT-based Cohesion Analysis of Japanese Texts (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in neural networks have significantly improved natural language processing tasks . they include self training-based language models such as BERT . |
| Approach: | They tackle a systematic analysis of cohesion in Japanese texts using BERT models . they find that coreference resolution is different in nature from other tasks . |
| Outcome: | The proposed analysis outperforms existing studies on cohesion in Japanese texts. |
Self-Attentional Models for Lattice Inputs (P19-1)
Copied to clipboard
| Challenge: | Existing work has extended recurrent neural networks to model lattice inputs but these models suffer from slow computation speeds. |
| Approach: | They propose to extend the paradigm of self-attention to handle lattice inputs by adding probabilistic reachability masks that incorporate latticae structure into the model and support lattics if available. |
| Outcome: | The proposed model outperforms baseline models while being much faster to compute than previous models. |
MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for training pre-trained language models have limited practicality due to latency requirements. |
| Approach: | They propose a method that uses a Mixture-of-Experts structure to increase model capacity and inference speed. |
| Outcome: | The proposed method outperforms existing distillation methods on natural language understanding and question answering tasks. |
Enhancing One-Shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism (2025.coling-main)
Copied to clipboard
Guanchen Li, Xiandong Zhao, Lian Liu, Zeping Li, Yixing Xu, Dong Li, Lu Tian, Jie He, Ashish Sirasao, Emad Barsoum
| Challenge: | Pre-trained language models (PLMs) are robust in contextual understanding but their considerable size incurs significant computational and storage costs. |
| Approach: | They propose a Sparse-Dense-Sparse pruning framework to prune PLMs . they prune less critical connections using conventional pruning methods . |
| Outcome: | The proposed pruning framework outperforms SparseGPT and Wanda under identical sparsity. |
Improving Constituent Representation with Hypertree Neural Networks (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods of span representation are based on simple derivations from word representations and do not utilize compositional structures of natural language. |
| Approach: | They propose a hypertree neural network that is structured with constituency parse trees to improve representations of constituent spans. |
| Outcome: | The proposed model improves representations of constituent spans using constituency parse trees. |
Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words (2022.coling-1)
Copied to clipboard
Haochun Wang, Chi Liu, Nuwa Xi, Sendong Zhao, Meizhi Ju, Shiwei Zhang, Ziheng Zhang, Yefeng Zheng, Bing Qin, Ting Liu
| Challenge: | Pre-trained models perform poorly with limited data and rare biomedical words. |
| Approach: | They propose to use prompt to fine-tune pre-trained models for biomedical domain tuning with a simple approach. |
| Outcome: | The proposed method achieves up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings. |
Distilling Knowledge for Search-based Structured Prediction (P18-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy. |
| Approach: | They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space. |
| Outcome: | The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines. |
MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, Na Zou
| Challenge: | Existing RAG systems struggle with the quality of retrieval documents, causing performance degradation and reducing performance. |
| Approach: | They propose a training-free RAG framework that leverages multiple LLM agents to collaboratively filter and score retrieved documents. |
| Outcome: | The proposed framework outperforms existing RAG frameworks in QA benchmarks and shows superior answer consistency and answer accuracy over baseline methods. |
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used in real-world industrial scenarios for various natural language processing tasks, but their high inference cost makes their deployment impractical, necessitating the use of smaller models. |
| Approach: | They propose a continual pre-training technique that generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. |
| Outcome: | The proposed technique improves small LLMs’ domain adaptability for business conversational tasks, compared with traditional methods that rely on next-token prediction. |
Original Content Is All You Need! an Empirical Study on Leveraging Answer Summary for WikiHowQA Answer Selection Task (2022.coling-1)
Copied to clipboard
Liang Wen, Juan Li, Houfeng Wang, Yingwei Luo, Xiaolin Wang, Xiaodong Zhang, Zhicong Cheng, Dawei Yin
| Challenge: | Existing answer selection approaches for community question answering lack additional answer summaries due to redundancy and lengthiness issues of crowdsourced answers. |
| Approach: | They constructed a dataset which contains a corresponding reference summary for each original lengthy answer. |
| Outcome: | The proposed model improves the performance of a question and candidate answer on a WikiHowQA dataset. |
Trucidator: Document-level Event Factuality Identification via Hallucination Enhancement and Cross-Document Inference (2025.coling-main)
Copied to clipboard
| Challenge: | Document-level event factuality identification (DEFI) assesses the veracity degree to which an event mentioned in a document has happened. |
| Approach: | They propose a document-level event factuality identification framework with hallucination features . they propose factualusion corpus that integrates both genuine and hallucinous false information . |
| Outcome: | The proposed framework outperforms baselines in document event factuality identification. |
Enhancing Chinese Pre-trained Language Model via Heterogeneous Linguistics Graph (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained Chinese language models ignore linguistics knowledge to learn representations. |
| Approach: | They propose a task-free enhancement module to integrate linguistics knowledge into Chinese pre-trained language models. |
| Outcome: | The proposed model improves Chinese pre-trained language models on 6 tasks with 10 benchmark datasets. |
Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work focuses on learning deep NER models with weak supervision without any human annotation. |
| Approach: | They propose a framework that can suppress the noise of the weak labels and fine-tune over the strongly labeled data. |
| Outcome: | The proposed framework outperforms existing methods on Named Entity Recognition tasks with weak supervision and weakly labeled data. |
GenSense: A Generalized Sense Retrofitting Model (C18-1)
Copied to clipboard
| Challenge: | Existing word embedding models use only one vector to represent a word, which is problematic in some natural language processing tasks that require sense level representation. |
| Approach: | They propose a generalized sense embedding learning framework that integrates with the semantic relations between the senses, the relation strength and the semantic strength. |
| Outcome: | The proposed model outperforms previous models in three types of experiments: semantic relatedness, contextual word similarity and semantic difference. |
PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models excel in various tasks, but their huge size and inaccessibility of parameters present challenges for practical deployment. |
| Approach: | They propose to use CoT data to distill task-specific ability from large language models to smaller models . they use reasoning programs to suppress errors in distilled data and improve distillation quality . |
| Outcome: | The proposed model outperforms LLMs on arithmetic reasoning, symbolic reasoning, and general ability. |
When Format Changes Meaning: Investigating Semantic Inconsistency of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are vulnerable to semantic inconsistency, a study finds . minor formatting variations result in divergent predictions for semantically equivalent inputs. |
| Approach: | They evaluate LLMs for semantic inconsistency and find they remain vulnerable . they propose to use mechanistic analysis to develop models that improve their reliability . |
| Outcome: | The proposed model is vulnerable to semantic inconsistency, the authors show . their model is brittle even in state-of-the-art models, they say . |
LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, but their proficiency in mathematical reasoning remains a key challenge. |
| Approach: | They propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
| Outcome: | The proposed framework evaluates LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated considerable success in various natural language processing tasks, but their performance in NMT tasks is still underexplored. |
| Approach: | They propose to use LLMs as automatic post-editors rather than direct translators to improve BLEU and COMET performance. |
| Outcome: | The proposed approach improves BLEU but COMET performance compared to in-context learning. |
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing efforts to train pre-trained language models have brought significant improvements to various NLP applications. |
| Approach: | They propose to compress bulky LMs while preserving useful information for a specific task. |
| Outcome: | The proposed method can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. |
Self-Attention with Cross-Lingual Position Representation (2020.acl-main)
Copied to clipboard
| Challenge: | Position encoding (PE) is used to preserve word order information for natural language processing tasks, generating fixed position indices for input sequences. |
| Approach: | They propose to augment SANs with cross-lingual position representations to model bilingually aware latent structure for the input sentence. |
| Outcome: | The proposed model significantly improves translation quality over baselines on EnglishGerman, JapaneseEnglish, and ChineseEnglish translation tasks. |
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024.acl-long)
Copied to clipboard
Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks, but their application to information retrieval tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. |
| Approach: | They propose to use instruction tuning to enhance LLMs' proficiency in IR tasks by combining a dataset with manually written templates to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions. |
| Outcome: | The proposed model can be used to perform query understanding, document understanding, and query-document relationship understanding tasks. |
TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced tasks like text summarization, but their size and computational demands limit their use in resource-constrained and privacy-centric settings. |
| Approach: | They propose a framework for distilling LLMs’ text summarization abilities into a compact, local model using a curriculum learning strategy that evolves from simple to complex tasks. |
| Outcome: | The proposed framework outperforms baseline models on CNN/DailyMail, XSum, and ClinicalTrial, and improves interpretability by providing insights into the summarization rationale. |
Automated Essay Scoring System for Nonnative Japanese Learners (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing systems only provide a holistic score that summarizes the quality of an essay, which provides little feedback for a language learner. |
| Approach: | They developed an automated essay scoring system for Japanese as a second language learners using an essay dataset with annotations for a holistic score and multiple trait scores. |
| Outcome: | The proposed system achieves the highest accuracy in various natural language processing tasks. |
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering (2023.acl-short)
Copied to clipboard
| Challenge: | Existing dense retrieval models are parameter-inefficient and underperform sparse counterparts. |
| Approach: | They propose a task-aware specialization for dEnse Retrieval architecture that enables parameter sharing by interleaving shared and specialized blocks in a single encoder. |
| Outcome: | The proposed architecture surpasses BM25 on questions and passages using 60% of the parameters as bi-encoder dense retrievers. |
Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation? (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pre-training masked language models with artificial data has been proven beneficial for several natural language processing tasks, however, it has been less explored for neural machine translation (NMT). |
| Approach: | They pre-trained masked language models with random sequences and created artificial data mimicking token frequency information from the real world. |
| Outcome: | The results show that pre-training models with artificial data improves translation performance in low-resource situations. |
Large Language Models are Better Reasoners with Self-Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve complex natural language processing tasks require multiple steps to verify the answers. |
| Approach: | They propose to use chain of thought prompting to solve reasoning tasks with large language models. |
| Outcome: | The proposed method can improve reasoning performance on arithmetic, commonsense, and logical reasoning datasets. |
PRIME: Ultra-Low-Rank Principal–Residual Model Merging (2026.findings-acl)
Copied to clipboard
Seung-Ho Lee, Kyungsu Lee, Bazarvaani Zuchi, Jeongmin Ahn, Insuk Seo, Donghyeon Jeon, Inho Kang, Seung-Hoon Na
| Challenge: | Existing methods for model merging have been limited by task-specific performance and task-related tasks. |
| Approach: | They propose an ultra-low-rank principal-residual model merging framework that decomposes task vector merging into two stages. |
| Outcome: | Experiments on eight natural language processing tasks show that PRIME outperforms existing models while preserving the task-specific capabilities of the original models. |
Task-oriented Word Embedding for Text Classification (C18-1)
Copied to clipboard
| Challenge: | Existing word embeddings only consider contextual information, which is suboptimal when used in various tasks due to a lack of task-specific features. |
| Approach: | They propose a task-oriented word embedding method that regularizes the distribution of words to enable a clear classification boundary. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on a text classification task. |
Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis (2022.acl-long)
Copied to clipboard
| Challenge: | Existing prompt tuning methods for cross-domain sentiment analysis have been underutilized due to domain discrepancy in the token distributions. |
| Approach: | They propose a new method to model cross-domain sentiment analysis using pre-trained language models by using soft prompts instead of hard templates. |
| Outcome: | The proposed method achieves state-of-the-art results on a publicly available sentiment analysis dataset. |
Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for data-driven annotations require domain-specific and task-aligned supervision. |
| Approach: | They propose a multi-label and multi-target sampling strategy to optimize the annotation quality. |
| Outcome: | The proposed method significantly improves performance and learning efficacy on the benchmark stance detection corpora. |
On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs . |
| Approach: | They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models . |
| Outcome: | The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks. |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |
A Web Portal about the State of the Art of NLP Tasks in Spanish (2024.lrec-main)
Copied to clipboard
Enrique Amigó, Jorge Carrillo-de-Albornoz, Andrés Fernández, Julio Gonzalo, Guillermo Marco, Roser Morante, Laura Plaza, Jacobo Pedrosa
| Challenge: | a web portal has been created with information about the state of the art of natural language processing tasks in Spanish. |
| Approach: | They propose a web portal that provides information about the state of the art of natural language processing tasks in Spanish. |
| Outcome: | The portal provides information about forums, competitions, tasks and datasets in Spanish that would otherwise be spread in multiple articles and web sites. |
Multi-pretraining for Large-scale Text Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for large-scale text classification involve excessive computation and memory overheads. |
| Approach: | They propose a self-supervised and weakly supervised pretraining frameworks for large-scale text classification with multiple categories. |
| Outcome: | The proposed framework improves on the self-supervised and weakly supervised methods while being computationally efficient. |
A Structured Span Selector (2022.naacl-main)
Copied to clipboard
| Challenge: | a typical approach to natural language processing tasks involves selecting text spans and making decisions about them. |
| Approach: | They propose a grammar-based structured span selection model which learns to make use of partial span annotations. |
| Outcome: | The proposed model improves on two popular span prediction tasks. |
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a promising solution for large language models, but their deployment remains computationally expensive. |
| Approach: | They propose a framework which iteratively balances training data within a fixed computational budget and enables the transfer of knowledge from expensive teacher LLMs to smaller student models. |
| Outcome: | The proposed framework achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models. |
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval (2021.findings-acl)
Copied to clipboard
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, Ji-Rong Wen
| Challenge: | Recent studies only consider query-centric similarity relation when learning the dual-encoder retriever. |
| Approach: | They propose a query-centric and PAssage-centric approach to capture more comprehensive similarity relations for dense passage retrieval. |
| Outcome: | The proposed approach significantly outperforms existing models on both MSMARCO and Natural Questions datasets. |
Gender Bias in Masked Language Models for Multiple Languages (2022.naacl-main)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. |
| Approach: | They propose to use English attribute word lists to evaluate bias in eight languages without manually annotating data. |
| Outcome: | The proposed model significantly correlates with the existing English datasets for gender bias. |
SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work shows promising results when prompting pre-trained language models, but in low-resource domains, the domain gap between the pre-training data and the downstream task is too large. |
| Approach: | They propose a method for prompting pre-trained language models using domain-specific keywords with a trainable gated prompt. |
| Outcome: | The proposed prompting method outperforms state-of-the-art prompting methods on three text classification benchmarks and shows that it reduces the need for domain-specific language model pre-training. |
Lexical Normalization for Code-switched Data and its Effect on POS Tagging (2021.eacl-main)
Copied to clipboard
| Challenge: | Social media data can be used to improve natural language processing performance, but it is often overlooked by lexical normalization systems. |
| Approach: | They propose three lexical normalization models specifically designed to handle code-switched data and evaluate their performance on POS tags. |
| Outcome: | The proposed models outperform monolingual models and lead to 5.4% performance increase for POS tagging compared to unnormalized input. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
Don’t Generate, Classify! Low-Latency Prompt Optimization with Structured Complementary Prompt (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated strong performance across diverse tasks, but their performance varies significantly across different prompts. |
| Approach: | They propose a framework that reframes prompt engineering as a classification problem. |
| Outcome: | The proposed framework improves answer quality by up to 26.5% in win rate compared to prior methods while reducing latency by upto 1,956 times. |
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, outperforming small PLMs by a large margin. |
| Approach: | They propose to scale up parameters of pre-trained language models only during fine-tuning to benefit from over-parameterization. |
| Outcome: | The proposed approach can significantly boost the fine-tuning performance of small PLMs and even help small PDMs outperform 3 parameterized larger ones. |
What do Large Language Models Need for Machine Translation Evaluation? (2024.emnlp-main)
Copied to clipboard
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, Fred Blain
| Challenge: | Existing research shows that large language models can perform better in machine translation tasks. |
| Approach: | They propose to use large language models for machine translation evaluations . authors explore what translation information is needed for LLMs to evaluate MT quality . |
| Outcome: | The proposed model performs comparable to fine-tuned multilingual pre-trained models. |
Probing Pre-trained Language Models for Semantic Attributes and their Values (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PTLMs) are used for many tasks including syntax, semantics and commonsense. |
| Approach: | They propose to integrate semantic attributes and their values into pretrained language models to improve their performance on many natural language processing tasks. |
| Outcome: | The proposed model performs better on masked tokens than humans on this task. |
BLN600: A Parallel Corpus of Machine/Human Transcribed Nineteenth Century Newspaper Texts (2024.lrec-main)
Copied to clipboard
| Challenge: | Historical documents present unique challenges to automated digital transcription technologies, such as optical character recognition (OCR). |
| Approach: | They propose to use a publicly available nineteenth-century newspaper corpus to train and develop OCR and post-OCR correction methodologies for historical newspaper machine transcription. |
| Outcome: | The proposed corpus will be useful for training and development of OCR and post-OCR correction methodologies for historical newspaper machine transcription. |
Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for neural machine translation (NMT) use encoder-only enhancement or rely on specific multilingual PLMs. |
| Approach: | They propose a monolingual PLM-sponsored NMT model that lets both encoder and decoder enjoy PLM enhancement. |
| Outcome: | The proposed model improves on IWSLT14 En-De, De-En, WMT14 En -De and En-Fr tasks and achieves new state-of-the-art. |
Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning (2024.acl-long)
Copied to clipboard
Zeqi Tan, Yongliang Shen, Xiaoxia Cheng, Chang Zong, Wenqi Zhang, Jian Shao, Weiming Lu, Yueting Zhuang
| Challenge: | Large language models (LLMs) have shown remarkable performance, but their training costs are exorbitant. |
| Approach: | They propose a parameter-efficient method for exploring optimal solutions within latent space by using latent units to extract input representations from LLMs. |
| Outcome: | The proposed method improves performance on a range of natural language processing tasks. |
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that passage retrieval and passage reranking are important for achieving mutual improvement. |
| Approach: | They propose a unified listwise training approach for passage retrieval and passage reranking that incorporates a retrieval procedure and a hybrid data augmentation strategy. |
| Outcome: | The proposed approach improves on both MSMARCO and Natural Questions datasets. |
Edit-Wise Preference Optimization for Grammatical Error Correction (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been successful in grammatical error correction (GEC) but their strengths have yet to be fully demonstrated in GEC . |
| Approach: | They propose a method to optimize grammatical errors by assigning higher reward weights to edit tokens during preference optimization. |
| Outcome: | The proposed method outperforms baselines on English and Chinese datasets and achieves state-of-the-art performance. |
Instruction-following Evaluation through Verbalizer Manipulation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on common instructions that align well with what the model learned during training, but proficiency in responding to these instructions does not necessarily imply strong ability in instruction following. |
| Approach: | They propose a new instruction-following evaluation protocol called verbalizer manipulation that instructs the model to verbalize the task label with words aligning with model priors to different extents. |
| Outcome: | The proposed protocol can be integrated with any classification benchmark to examine the model’s reliance on priors and its ability to override them to accurately follow the instructions. |
Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing literature has focused on pretrainer-based text-driven brain encoding models . however, few studies have explored the efficacy of task-specific learning of Transformers . |
| Approach: | They propose to use ten popular natural language processing tasks to learn Transformer representations for predicting brain responses. |
| Outcome: | The proposed model predicts brain activity across the whole brain. |
Prompting Few-shot Multi-hop Question Generation via Comprehending Type-aware Semantics (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches for multi-hop question generation rely on large annotated data . supervised approaches rely only on large labeled data, making it hard to perform tasks. |
| Approach: | They propose a type-aware semantics extraction-based chain-of-thought method for multi-hop question generation for documents . they first extract question types and essential semantic phrases from the given documents and the answer . |
| Outcome: | The proposed approach extracts question types and essential semantic phrases from documents and the answer. |
Transformer-Exclusive Cross-Modal Representation for Vision and Language (2021.findings-acl)
Copied to clipboard
| Challenge: | a number of approaches to crossmodal representation have been used, but transformer architecture has taken over the recurrent neural networks in natural language processing tasks. |
| Approach: | They propose to use transformer architecture to handle cross-modal representations for vision and language with compatible performance to convolutional neural networks. |
| Outcome: | The proposed model outperforms recurrent neural networks in vision and language representations with transformer architecture. |
RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained language models exhibit poor generalization and robustness in adversarial settings. |
| Approach: | They propose a self-supervised sentence embedding framework that improves generalization and robustness against adversarial attacks. |
| Outcome: | The proposed framework reduces the success rate of adversarial attacks by almost half . it also improves semantic text similarity tasks and various transfer tasks . |
Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations (C18-1)
Copied to clipboard
| Challenge: | Neural activation models have been proposed to map word semantics to localized neural activations. |
| Approach: | They propose a computational model that estimates semantic similarity in the neural activation space and investigate its performance for various natural language processing tasks. |
| Outcome: | The proposed model performs better than state-of-the-art word embeddings for the task of semantic similarity estimation between very similar or very dissimilar words while performing well on other tasks such as entailment and word categorization. |
schuBERT: Optimizing Elements of BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer based models have achieved state-of-the-art performance for many natural language processing tasks including machine translation, question-answering tasks and semantic role labeling. |
| Approach: | They propose to reduce the number of parameters of BERT to obtain a much efficient light model. |
| Outcome: | The proposed model achieves 6.6% higher average accuracy on GLUE and SQuAD datasets than the previous model with three encoder layers while having the same number of parameters. |
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown overwhelming performances on many tasks, but their large size and slow inference speed have hindered practical deployments. |
| Approach: | They propose a hierarchical relational knowledge distillation method to capture hierarchic and domain relational information. |
| Outcome: | The proposed method outperforms existing methods on multi-domain datasets and is highly reproducible. |
Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that pretrain for cross-lingual tasks do not improve cross-linguistic learning. |
| Approach: | They propose to employ machine translation as a continued training objective to enhance language representation learning by bridging multilingual pretraining and cross-lingual applications. |
| Outcome: | The proposed model performance is compared with existing models and their latent representations. |
Using J-K-fold Cross Validation To Reduce Variance When Tuning NLP Models (C18-1)
Copied to clipboard
| Challenge: | a recent study shows that performance estimations are unstable and variable . this makes it difficult to use parameter tuning and model selection . |
| Approach: | They propose to use a less variable CV method to evaluate performance . they propose lower choices of K than are typically seen in the NLP literature . |
| Outcome: | The proposed method can be used for parameter tuning and performance estimation, but it is unstable and unstable. |
SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at various tasks but are vulnerable to jailbreak attacks that induce harmful content generation. |
| Approach: | They propose a reinforcement learning framework that leverages the model’s own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. |
| Outcome: | The proposed framework improves model safety by iterative self-improvement without additional annotated data or external models during training phase. |
An Empirical Investigation of Error Types in Vietnamese Parsing (C18-1)
Copied to clipboard
| Challenge: | Syntactic parsing improves the quality of natural language processing tasks. |
| Approach: | They evaluated Vietnamese Treebank model to find most suitable parsing method . they found that Vietnamese parsers produced limited training data and POS errors . |
| Outcome: | The proposed method improves the parsing quality in Vietnamese . the results highlight three possible sources of parser errors . |
Incorporating Attribution Importance for Improving Faithfulness Metrics (2023.acl-long)
Copied to clipboard
| Challenge: | Feature attribution methods (FAs) are popular for providing insights into the model reasoning process of making predictions. |
| Approach: | They propose a simple yet effective criterion that randomly masks tokens proportionately to their FA importance. |
| Outcome: | The proposed method is more faithful than hard sufficiency and comprehensiveness metrics. |
Uncertainty Guided Global Memory Improves Multi-Hop Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformers are used to solve multi-hop question answering tasks that require reasoning over multiple parts of a long document. |
| Approach: | They propose a method that collects relevant information over the entire document and then combines it with local context to solve a multi-hop question answering task. |
| Outcome: | The proposed method improves on three MHQA datasets compared to the baseline model. |
M-BRe: Discovering Training Samples for Relation Extraction from Unlabeled Texts with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract training instances from unlabeled texts are expensive . sentences that contain the target relations in texts can be scarce and difficult to find . |
| Approach: | They propose a framework that can automatically extract training instances from unlabeled texts for RE. |
| Outcome: | The proposed method can extract training instances from unlabeled texts for RE. |
A Survey on Automatically-Constructed WordNets and their Evaluation: Lexical and Word Embedding-based Approaches (L18-1)
Copied to clipboard
| Challenge: | WordNets are lexical databases in which groups of synonyms are stored according to the semantic relationships between them. |
| Approach: | This paper describes various approaches to constructing WordNets automatically by leveraging traditional lexical resources and newer trends such as word embeddings. |
| Outcome: | The proposed methods leverage traditional lexical resources and newer trends such as word embeddings to build and evaluate WordNets. |
Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are renowned for their extensive linguistic knowledge and strong generalization capabilities, but their high computational demands make them unsuitable for resource-constrained environments. |
| Approach: | They propose a framework that integrates a single frozen layer from an LLM into a SLM and fine-tunes the combined model for specific tasks. |
| Outcome: | The proposed framework improves performance across a range of natural language processing tasks, including both natural language understanding and generation. |
Character-Level Feature Extraction with Densely Connected Networks (C18-1)
Copied to clipboard
| Challenge: | Existing methods to generate character-level features with neural architectures such as CNN or Recurrent Neural Network (RNN) are slow and generate position-independent features. |
| Approach: | They propose a method that uses a densely connected network to extract character-level features from words using CNN and RNN. |
| Outcome: | The proposed method shows robustness and effectiveness while being faster than CNN- or RNN-based methods. |
Enhancing Contextual Word Representations Using Embedding of Neighboring Entities in Knowledge Graphs (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for pre-trained language models lack explicit grounding in real-world entities. |
| Approach: | They propose a mechanism that integrates the structure of a KG into recent PLM architectures by generalizing the embeddings of neighboring entities. |
| Outcome: | The proposed method improves a classification task, entity typing task and language comprehension tasks. |
Sentence Matching with Syntax- and Semantics-Aware BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Sentence matching aims to determine the special relationship between two sentences. |
| Approach: | They propose to integrate syntactic and semantic information into BERT with sentence matching by using an implicit integration method that is less sensitive to the output structure information. |
| Outcome: | The proposed method achieves state-of-the-art or competitive performance on several sentence matching datasets. |
iParaphrasing: Extracting Visually Grounded Paraphrases via an Image (C18-1)
Copied to clipboard
| Challenge: | iParaphrasing extracts visually grounded paraphrases, which are different phrasal expressions describing the same visual concept in an image. |
| Approach: | They propose a task to extract visually grounded paraphrases from images . they propose to model the similarity between the extracted VGPs using existing methods . |
| Outcome: | The proposed task extracts visually grounded paraphrases from images . the proposed method has the potential to improve multimodal language and image tasks . |
TINA: Textual Inference with Negation Augmentation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing transformer-based models perform poorly on textual entailment when examples contain negations. |
| Approach: | They propose a new definition of textual entailment that captures negation and a principled technique for negated data augmentation that can be combined with unlikelihood loss function. |
| Outcome: | The proposed method significantly improves on textual entailment datasets with negations without sacrificing performance on datasets without negation. |
ResearchArena: Benchmarking Large Language Models’ Ability to Collect and Organize Information as Research Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel across many natural language processing tasks but face challenges in domain-specific, analytical tasks such as conducting research surveys. |
| Approach: | They propose a benchmark to evaluate LLMs' capabilities in conducting research surveys. |
| Outcome: | The proposed benchmark is designed to evaluate LLMs' capabilities in conducting research surveys. |
Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work on data augmentation techniques that interpolate inputs and labels shows strong effectiveness in image classification. |
| Approach: | They propose to integrate mixup to transformer-based pre-trained architecture for NLP tasks while keeping the whole end-to-end training system. |
| Outcome: | The proposed framework improves on GLUEbenchmark and transformer-based learning models while keeping the whole end-to-end training system. |
RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing PLMs suffer from poor robustness in adversarial scenarios, despite their success with unseen samples. |
| Approach: | They propose a self-supervised sentence embedding framework that enhances generalization and robustness in various text representation tasks and against diverse adversarial attacks. |
| Outcome: | The proposed framework improves generalization and robustness in various representation tasks and against diverse adversarial attacks. |
CoLT5: Faster Long-Range Transformers with Conditional Computation (2023.emnlp-main)
Copied to clipboard
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
| Challenge: | Many natural language processing tasks require long inputs, but processing long documents with a Transformer model is expensive due to quadratic attention complexity and applying feedforward and attention projection layers to every input token. |
| Approach: | They propose a long-input Transformer model that builds on the intuition that some tokens are more important than others and uses conditional computation to devote more computation to important tokens. |
| Outcome: | The proposed model achieves stronger performance than LongT5 with faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. |
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks. |
| Approach: | They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection . |
| Outcome: | The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters. |
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang
| Challenge: | Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored. |
| Approach: | They propose to use a benchmark to evaluate large language models' financial domain knowledge and practical abilities. |
| Outcome: | The proposed benchmark evaluates large language models' financial domain knowledge and practical abilities. |
Continual Learning for Sentence Representations Using Conceptors (N19-1)
Copied to clipboard
| Challenge: | Existing sentence encoders for distributed representations of sentences are limited in their performance on fixed corpora. |
| Approach: | They propose a continual learning scenario for distributed representations of sentences . they initialize sentence encoders with corpus-independent features and update them sequentially . |
| Outcome: | The proposed sentence encoder can learn features from new corpora while maintaining its competence on previously encountered corporales. |
TasTe: Teaching Large Language Models to Translate through Self-Reflection (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to enhance LLMs' performance in machine translation are unable to fully exploit their instruction-following capabilities. |
| Approach: | They propose a framework for translating through self-reflection that involves two stages of inference . they propose to use the framework to refine LLMs' preliminary translations . |
| Outcome: | The proposed framework can produce translation outputs that match the quality of NMT systems. |
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on parametric models that store knowledge in parameters or retrieval-augmented models that have access to external knowledge sources. |
| Approach: | They propose a parametric parametric model that stores knowledge in its parameters or a retrieval-augmented model that has access to external knowledge sources. |
| Outcome: | The proposed method runs substantially faster across the board and produces more accurate results on WoW and ELI5. |
ARCH: Efficient Adversarial Regularized Training with Caching (2021.findings-emnlp)
Copied to clipboard
Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
| Challenge: | Existing approaches to regularize models require generating a perturbation for each sample in each epoch. |
| Approach: | They propose an adversarial regularization method where perturbations are generated and cached once every several epochs. |
| Outcome: | The proposed method significantly eases the computational burden (saves up to 70% of computational time) it produces a notably better (in most of the tasks) or comparable model generalization. |
No Data to Crawl? Monolingual Corpus Creation from PDF Files of Truly low-Resource Languages in Peru (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for extracting text from PDF files are expensive and limited by the absence of web content of endangered languages. |
| Approach: | They propose a method for creating monolingual corpora for four endangered languages . they use a PDF file format with multilingual sentences and noisy pages . |
| Outcome: | The proposed method allows the creation of clean corpora for the four languages, a key resource for natural language processing tasks nowadays. |
Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration (2025.naacl-long)
Copied to clipboard
Ang Li, Jingqian Zhao, Bin Liang, Lin Gui, Hui Wang, Xi Zeng, Xingwei Liang, Kam-Fai Wong, Ruifeng Xu
| Challenge: | Large language models generate biased stances due to spurious correlations and preference towards certain individuals and topics. |
| Approach: | They propose a counterfactual Augmented Calibration Network to calibrate potential bias in stance detection of large language models. |
| Outcome: | The proposed calibration network can mitigate biases of large language models, achieving state-of-the-art results. |
KinyaBERT: a Morphology-aware Kinyarwanda Language Model (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT are sub-optimal at handling morphologically rich languages. |
| Approach: | They propose a two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphology in a low-resource Kinyarwanda language. |
| Outcome: | The proposed model outperforms baseline models on the low-resource morphologically rich Kinyarwanda language by 2% in F1 score and 4.3% in average score of GLUE benchmark. |
Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for analyzing argumentative debates are insufficient to understand complex tasks. |
| Approach: | They propose a hybrid method to automatically predict the winning stance in argumentative debates using arguments from argumentation theory and semantics. |
| Outcome: | The proposed method is based on an unexplored new instance of the automatic analysis of natural language arguments. |
Mask-Align: Self-Supervised Neural Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Word alignment is an important task in many natural language processing tasks. |
| Approach: | They propose a self-supervised word alignment model that takes advantage of the full context on the target side. |
| Outcome: | The proposed model outperforms previous unsupervised models and obtains state-of-the-art results on four language pairs. |
TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices. |
| Approach: | They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages. |
| Outcome: | The proposed method accelerates inference and reduces model size while maintaining accuracy. |
Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for word embedding compression are limited . word embeds have a considerable size and need to be compressed to deploy on edge devices . |
| Approach: | They propose a block-wise low-rank approximation method for word embedding called GroupReduce . they propose 'frequency-inverse document frequency method' and a differentiable method for weighting . |
| Outcome: | The proposed algorithm more effectively finds word weights than competitors in most cases. |
Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction (2021.findings-acl)
Copied to clipboard
| Challenge: | Contextualized word representations are effective in many natural language processing tasks, but it remains unclear to what extent they can cover hand-coded semantic information such as semantic frames. |
| Approach: | They compare contextualized word representations with two English frame-semantic resources . they find that several contextualized representations are informative for semantic frame induction . |
| Outcome: | The proposed representations are useful in natural language processing tasks, but are not fully understood by the literature. |
Effectiveness of French Language Models on Abstractive Dialogue Summarization Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models have established the state-of-the-art on various natural language processing tasks, including dialogue summarization. |
| Approach: | They propose to use several language specific pre-trained models to summarize spontaneous oral dialogues in French using several language-specific pre-trainers: BARThez, BelGPT-2, mBARThes, and mT5. |
| Outcome: | The proposed models outperform the existing models on the DECODA (Call Center) dialogue corpus and show that they are far superior to the current models. |
Intermediate Self-supervised Learning for Machine Translation Quality Estimation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for machine translation quality estimation (QE) rely on annotated data. |
| Approach: | They propose a self-supervised learning task for machine translation (MT) that orients a pre-trained model towards the target task. |
| Outcome: | The proposed method outperforms existing methods on English-to-German and English- to-Russian translation directions and is comparable to existing models. |
Text Summarization with Pretrained Encoders (D19-1)
Copied to clipboard
| Challenge: | Existing pretraining languages such as ELMo and GPT have advanced a wide range of tasks. |
| Approach: | They propose a novel document-level encoder based on BERT which can express the semantics of a document and obtain representations for its sentences. |
| Outcome: | The proposed model achieves state-of-the-art in extractive and abstractive settings. |
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
BRCC and SentiBahasaRojak: The First Bahasa Rojak Corpus for Pretraining and Sentiment Analysis Dataset (2022.coling-1)
Copied to clipboard
| Challenge: | Code-mixing is prevalent in multilingual societies and is challenging to train . we use data augmentation to build a model to deal with code-mixed inputs . |
| Approach: | They propose to train a model to deal with code-mixing phenomena of Bahasa Rojak using data augmentation to construct a Bahasan Rojakin corpus and a pre-trained model to process input tokens. |
| Outcome: | The proposed model can tag the language of the input token automatically to process code-mixing input. |
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing BERT-based pre-trained language models achieve high performance on many downstream tasks, but native derived sentence representations are collapsed and thus poor performance on semantic textual similarity (STS) tasks. |
| Approach: | They propose a framework for self-supervised Sentence Representation Transfer that adopts contrastive learning to fine-tune BERT in an unsupervised way. |
| Outcome: | The proposed framework improves on the BERT-derived representations by 8% on STS datasets and shows robustness in data scarcity scenarios. |
OFrLex: A Computational Morphological and Syntactic Lexicon for Old French (2020.lrec-1)
Copied to clipboard
| Challenge: | Using heterogeneous language resources, we extract structured and exploitable information from a large-coverage morphological and syntactic Old French lexicon. |
| Approach: | They propose to use a large-coverage morphological and syntactic Old French lexicon to extract structured and exploitable information from heterogeneous language resources. |
| Outcome: | The proposed extension technique will be validated manually in the near future and take advantage of OFrLex’s viewing, searching and editing interface. |
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve inference efficiency target to reduce per-layer latency, but ignore cumulative latency due to number of layers. |
| Approach: | They propose to identify quasi-independent layers that can be concurrently computed to significantly decrease inference latency. |
| Outcome: | Empirical results show that the proposed method reduces latency by 48.3% on LLaMA-33B while maintaining close level of performance. |
Enhancing Language Model Hypernetworks with Restart: A Study on Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | a comprehensive investigation into optimization strategies for hypernetworks remains lacking. |
| Approach: | They propose restart optimization strategies to improve hypernetworks' performance for language models. |
| Outcome: | The proposed restart strategy improves hypernetworks' performance for language models, compared to conventional deep neural networks. |
Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent benchmarks. |
| Approach: | They compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. |
| Outcome: | The proposed models outperform auto-regressive models in English, French and Spanish on 14 NER datasets. |
Token and Head Adaptive Transformers for Efficient Natural Language Processing (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT have shown significant accuracy improvements on various tasks, but their computational cost and memory footprint are prohibitive. |
| Approach: | They propose to extend Length Adaptive Transformer to extend the model to a token and head pruning scheme to optimize pruning efficiency. |
| Outcome: | The proposed model can compress and accelerate BERT-based models by fine-tuning and a token and head pruning scheme. |
ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks. |
| Approach: | They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy. |
| Outcome: | The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks. |
Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)
Copied to clipboard
| Challenge: | Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks. |
| Approach: | They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks. |
| Outcome: | The proposed method improves on standard semantic textual similarity and semantic search tasks. |
Signed Coreference Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sign Language Processing is based on linguistic theories of spoken languages and expect either speech or written text as input. |
| Approach: | They propose a new challenge for coreference modeling and Sign Language Processing to solve this problem. |
| Outcome: | The proposed models will be linguistically informed and can address the complexities of the challenge effectively. |
Leveraging Locality in Abstractive Text Summarization (2022.emnlp-main)
Copied to clipboard
Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Neural attention models have improved on many natural language processing tasks, but their quadratic memory complexity hinders their applications in long text summarization. |
| Approach: | They propose to use a restricted context to study locality in text summarization . they propose to employ a quadratic memory growth with respect to the input length . |
| Outcome: | The proposed model has better performance than baseline models with efficient attention modules. |
Fine-tuned Language Models are Continual Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models relies on intuition that most tasks can be described via natural language instructions. |
| Approach: | They propose that a model should be able to keep extending its knowledge without forgetting previous skills. |
| Outcome: | The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets. |
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021.emnlp-main)
Copied to clipboard
| Challenge: | Commonsense knowledge bases are mostly human-generated and reflect societal biases . a filtering-based approach can reduce the issues in both resources and models but leads to a performance drop . |
| Approach: | They propose a filtering-based approach to mitigating representational harms in ConceptNet and GenericsKB . they propose filtered-based approaches can reduce issues in both resources and models but leads to performance drop . |
| Outcome: | The proposed approach reduces issues in resources and models but leads to performance drop . the paper proposes a filtering-based approach that reduces biases but leaves room for future work . |
Emergent Communication Pretraining for Few-Shot Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | state-of-the-art models that rely on multilingual pretrained encoders achieve sample efficiency in downstream applications, but lack abundant amounts of unlabelled text. |
| Approach: | They propose a method to pretrain neural networks via emergent communication from referential games by grounding communication on images as a crude approximation of real-world environments. |
| Outcome: | The proposed method significantly improves machine translation in few-shot settings and provides an evaluation protocol to probe the properties of emergent languages ex vitro. |
Intra-Event and Inter-Event Dependency-Aware Graph Network for Event Argument Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not build dependency information among event argument roles . Existing methods do not learn the interactions between different roles based on event structure . |
| Approach: | They propose an intra-event and inter-e event dependency-aware graph network to model dependencies between roles . they use event structure as the fundamental unit to construct role dependencies within events . |
| Outcome: | The proposed model improves on the ACE05, RAMS, and WikiEvents datasets. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Detecting Loanwords in Emakhuwa: An Extremely Low-Resource Bantu Language Exhibiting Significant Borrowing from Portuguese (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing corpora in African languages reveal significant spelling inconsistencies, contributing to poor-quality textual data when encountered in written form. |
| Approach: | They propose a supervised method to identify loanwords in Portuguese . they employ traditional machine learning algorithms incorporating handcrafted features . |
| Outcome: | The proposed method achieves the F1-score of 93% in Emakhuwa, borrowed from Portuguese. |
Learning to Compress Prompt in Natural Language Formats (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work rely on compressing long contexts into soft prompts, but soft prompt compression encounters limitations in transferability . natural language (NL) prompts are incompatible with back-propagation, and NL prompts lack flexibility in imposing length constraints. |
| Approach: | They propose a framework that compresses long prompts into NL formatted Capsule Prompts. |
| Outcome: | The proposed framework reduces 81.4% of the original length, decreases inference latency up to 4.5x, and saves 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets. |
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings are becoming a ubiquitous component of natural language processing. |
| Approach: | They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains . |
| Outcome: | The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter. |
An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to augment labeled data are limited by the scarcity of labeles . a method called contextual data augmentation (CDA) can be used to augment labels . |
| Approach: | They propose a data augmentation method that generates labeled training instances using a pretrained language model. |
| Outcome: | The proposed method can improve the quality of augmented training data compared to the conventional method. |
Learning Constraints for Structured Prediction Using Rectifier Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Various natural language processing tasks require domain expertise to design good constraints. |
| Approach: | They propose a framework for learning constraints in a network of linear inequalities over the output variables. |
| Outcome: | The proposed framework can be used to learn constraints from data on natural language processing tasks. |
Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment (2023.acl-long)
Copied to clipboard
Tianshu Yu, Haoyu Gao, Ting-En Lin, Min Yang, Yuchuan Wu, Wentao Ma, Chao Wang, Fei Huang, Yongbin Li
| Challenge: | Existing speech-text pre-training methods are limited to one or two specific tasks, despite their success in speech-language processing tasks. |
| Approach: | They propose a temporal position prediction task to capture the speech-text alignment . they use a textual dialog pre-training task to generalize a response selection task . |
| Outcome: | The proposed model is superior in learning speech-text alignment and multi-turn dialog context. |
Translating away Translationese without Parallel Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Translated texts exhibit systematic linguistic differences compared to original texts in the same language, referred to as translationese . studies show translationeses have effects on various cross-lingual natural language processing tasks . |
| Approach: | They propose a translation-based style transfer approach that learns from monolingual data . they combine a self-supervised approach with an unsupervised approach . |
| Outcome: | The proposed method reduces translationese to a level of a random classifier after style transfer while preserving the content and fluency in the target original style. |
ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to integrate large language models and knowledge graphs with LLMs often ignore the rich cognitive potential inherent in KGs. |
| Approach: | They propose an observation-driven agent framework that integrates KG reasoning abilities via global observation and integrates it into the action and reflection modules. |
| Outcome: | The proposed framework improves on several datasets and achieves 12.87% and 8.9% accuracy improvements. |
SemRoDe: Macro Adversarial Training to Learn Representations that are Robust to Word-Level Attacks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to defend against word-level attacks have been limited. |
| Approach: | They propose a new approach called Semantic Robust Defence to enhance the robustness of language models by aligning the domains with a distance-based objective. |
| Outcome: | The proposed approach can be generalized across word embeddings, even when they share minimal overlap at both vocabulary and word-substitution levels. |
Unsupervised Concept Representation Learning for Length-Varying Text Similarity (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing document similarity approaches suffer from the information gap caused by context and vocabulary mismatches when comparing varying-length texts. |
| Approach: | They propose an unsupervised concept representation learning approach to address this issue . they propose a concept-based document matching method to leverage recognition of local phrase features . |
| Outcome: | The proposed method achieves a better F1 score than baseline models on real-world data sets. |
PcMSP: A Dataset for Scientific Action Graphs Extraction from Polycrystalline Materials Synthesis Procedure Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | 305 open access scientific articles are used for synthesis action graphs . lack of annotated data has hindered progress in this field . |
| Approach: | They propose to annotate Polycrystalline Materials Synthesis Procedures PcMSP from 305 open access scientific articles for the construction of synthesis action graphs. |
| Outcome: | The proposed dataset contains the synthesis sentences, entity mentions and intra-sentence relations extracted from the experimental paragraphs. |
Distinguishing Between Foreground and Background Events in News (2020.coling-main)
Copied to clipboard
| Challenge: | a new task is needed to distinguish between foreground and background events in news articles . |
| Approach: | They propose a task of distinguishing between foreground and background events in news articles . they also identify the general temporal position of background events relative to the foregoing period . |
| Outcome: | The proposed model achieves good performance on a dataset of news articles . |
DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising (2025.acl-long)
Copied to clipboard
| Challenge: | Existing adversarial defense methods for natural language processing still pose challenges to adversarials. |
| Approach: | They propose a novel adversarial defense method that incorporates a diffusion layer as a denoiser between the encoder and the classifier. |
| Outcome: | The proposed method improves over existing adversarial defense methods and achieves state-of-the-art performance against black-box and white-box adversarials. |
Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks (D18-1)
Copied to clipboard
| Challenge: | Existing gated recurrent networks have a vanishing gradient, allowing for more matrix transformations and less transparent functions. |
| Approach: | They propose an additionsubtraction twin-gated recurrent network (ATR) to simplify neural machine translation. |
| Outcome: | The proposed system is more transparent than LSTM/GRU due to the simplification. |
A Framework for Effective Invocation Methods of Various LLM Services (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are becoming a fundamental tool for various natural language processing tasks due to commercial reasons, the potential risk of misuse and expensive tuning cost. |
| Approach: | They propose a framework for constructing an effective LLM services invocation strategy that best meets task demands. |
| Outcome: | The proposed framework classifies existing methods into four categories: input abstraction, semantic cache, solution design, and output enhancement, which can be used separately or jointly during the invocation life cycle. |
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. |
| Approach: | They propose a method for converting multi-head attention into grouped-query attention with any compression ratio of KV heads. |
| Outcome: | The proposed method can compress up to 87.5% KV heads of LLaMA2-7B model and 75% Kv heads of Sheared-LLa MA-1.3B with acceptable performance degradation. |
DORE: A Dataset for Portuguese Definition Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Definition modelling (DM) is the task of automatically generating a dictionary definition of a specific word. |
| Approach: | They propose to create a dataset for definition modelling for Portuguese with 100,000 definitions and evaluate several deep learning based DM models on the dataset. |
| Outcome: | The proposed dataset will facilitate research and study of Portuguese in wider contexts. |
Speed Reading: Learning to Read ForBackward via Shuttle (D18-1)
Copied to clipboard
| Challenge: | LSTM-Shuttle uses human speed reading techniques to perform natural language processing tasks. |
| Approach: | They propose a model which uses human speed reading techniques to perform natural language processing tasks for accurate and efficient comprehension. |
| Outcome: | The proposed model predicts on IMDB, Rotten Tomatoes, AG, and Children’s Book Test datasets and goes backwards. |
ltzGLUE: Luxembourgish General Language Understanding Evaluation (2026.findings-acl)
Copied to clipboard
Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cedric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
| Challenge: | ltzGLUE is the first official NLU benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. |
| Approach: | They propose a new natural language understanding (NLU) benchmark for Luxembourgish based on the popular GLUE benchmark for English. |
| Outcome: | The proposed model performs well across many languages and is based on the GLUE benchmark for English. |
ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable in-context learning capabilities in various natural language processing tasks. |
| Approach: | They propose a novel approach ERA-CoT which aids LLMs in understanding context by capturing relationships between entities and supports the reasoning of diverse tasks through Chain-of-Thoughts (CoT). |
| Outcome: | The proposed method improves on GPT3.5 and previous SOTA prompting methods by an average of 5.1% compared to previous prompting approaches. |
Topic-DPR: Topic-based Prompts for Dense Passage Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research focused on optimizing a single prompt as a continuous prompt, but this approach leads to a semantic space collapse, preventing differentiation between relevant and irrelevant passages. |
| Approach: | They propose a dense passage retrieval model that uses topic-based prompts and propose 'positive and negative sampling strategies' to boost dense retrieval efficiency. |
| Outcome: | The proposed model surpasses state-of-the-art retrieval techniques and improves space uniformity. |
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. |
| Approach: | They compare the performance of a linear classifier based on word embeddings with a pre-trained language model, i.e., BERT, across a wide range of datasets and classification tasks. |
| Outcome: | The proposed method outperforms baselines in standard datasets with large training sets, but in settings with small training datasets it performs better. |
Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that learning multiple training objectives in a single model makes the learned language representation barely converge to the desired optimum. |
| Approach: | They propose a meta-learning-based adaptive sampler which learns latent sampling pattern on arbitrary pre-training objectives. |
| Outcome: | Empirical studies show that learning multiple objectives in a single model makes it difficult to achieve the desired optimum. |
GDA: Grammar-based Data Augmentation for Text Classification using Slot Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies suggest data augmentation approaches to resolve the low-resource problem in natural language processing tasks. |
| Approach: | They propose to use slot information to augment sentences using a set of injective relations between a sentence’s semantics and its syntactical structure to augment the dataset. |
| Outcome: | The proposed approach outperforms all other data augmentation methods by 19.38%. |
Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for named entity recognition tasks are difficult to interpret . authors present a general methodology for interpretable evaluation for named entities . |
| Approach: | They propose a general methodology for interpretable evaluation for named entity recognition task. |
| Outcome: | The proposed evaluation method enables researchers to interpret differences in models and datasets . it makes it easy for future researchers to run similar analyses and drive progress in this area . |
Effective Distillation of Table-based Reasoning Ability from LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on table-based reasoning distillation has focused on smaller models with limited performance. |
| Approach: | They propose a table-based reasoning distillation approach to distill LLMs into smaller models . their results show that a 220 million parameter model fine-tuned using distilled data improves performance . |
| Outcome: | The proposed model improves on a scientific table-to-text generation dataset and surpasses specific LLMs. |
Empirical Study of Zero-Shot NER with ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been a key component of natural language processing (NLP) . |
| Approach: | They propose to decompose the NER task into simpler subproblems by labels and propose a syntactic augmentation strategy to stimulate model's intermediate thinking. |
| Outcome: | The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets. |
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks. however, their extensive memory requirements present significant challenges for deployment in resource-constrained environments. |
| Approach: | They propose a training-free framework that achieves ultra-low equivalent bit-width KV cache quantization. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on TruthfulQA and LongBench. |
Flight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work has shown that pre-trained transformers obtain remarkable performance on many natural language processing tasks including automatic summarization. |
| Approach: | They propose to use transformers to generate multi-document summarization where the summary is explicitly conditioned on a user-given topic statement or question. |
| Outcome: | The proposed models perform well on four challenging summarization datasets from the general domain and one from consumer health. |
Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches for language models have overlooked the difficulty of training examples. |
| Approach: | They propose a framework that controls difficulty of training examples during pre-training by a tutor network. |
| Outcome: | The proposed framework outperforms state-of-the-art KD methods with student models on the GLUE benchmark. |
Large Language Model-Based Event Relation Extraction with Rationales (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for ERE rely on large language models, but they face limitations. |
| Approach: | They propose an LLM-based approach with rationales for the ERE task . LLMERE transforms ERE into a question-and-answer task that may have multiple answers . |
| Outcome: | Experimental results show that LLMERE improves over existing methods. |
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations but comes at a substantial training cost. |
| Approach: | They propose a concept-based curriculum masking method that evaluates the MLM difficulty of each token based on a carefully-designed linguistic difficulty criterion. |
| Outcome: | The proposed method significantly improves pre-training efficiency with the original BERT model at half the training cost. |
The Importance of Being Recurrent for Modeling Hierarchical Structure (D18-1)
Copied to clipboard
| Challenge: | Recent work shows that recurrent neural networks can implicitly capture hierarchical information when trained to solve common natural language processing tasks. |
| Approach: | They propose a convolutional sequence-to-sequence model that exploits hierarchical information implicitly. |
| Outcome: | The proposed model is recurrent and non-recurrent, and it can model hierarchical structure implicitly. |
Incorporating Linguistic Constraints into Keyphrase Generation (P19-1)
Copied to clipboard
| Challenge: | Existing keyphrase generation methods generate overlapping phrases (including sub-phrases or super-phrase) Existing methods are far from satisfactory for a wide range of natural language processing tasks. |
| Approach: | They propose a parallel Seq2Seq network with coverage attention to alleviate the overlapping phrase problem by integrating linguistic constraints of keyphrase into the basic Seq2-Sequeq network on the source side and employ the multi-task learning framework on the target side. |
| Outcome: | The proposed method outperforms the state-of-the-art CopyRNN on scientific datasets and is also more effective in news domain. |
Towards Distribution-shift Robust Text Classification of Emotional Content (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that supervised models are more robust to change in domain and distribution, but the decrease in performance due to the distribution shift is still a major issue for supervised systems. |
| Approach: | They propose to fine-tune supervised models on task-specific datasets to achieve out-of-distribution performance. |
| Outcome: | The proposed model outperforms all available models in distribution and out of distribution with only a few thousand training samples. |
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach (2021.emnlp-main)
Copied to clipboard
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao
| Challenge: | Existing approaches to adversarial regularization treat adversarials and defending players equally, which is undesirable because only the defending player contributes to the generalization performance. |
| Approach: | They propose a method which formulates adversarial regularization as a Stackelberg game and induces a competition between a leader and a follower. |
| Outcome: | The proposed method outperforms existing adversarial regularization baselines on a set of machine translation and natural language understanding tasks. |
Enhancing Few-Shot Topic Classification with Verbalizers. a Study on Automatic Verbalizer and Ensemble Methods (2024.lrec-main)
Copied to clipboard
Quang Anh Nguyen, Nadi Tomeh, Mustapha Lebbah, Thierry Charnois, Hanene Azzag, Santiago Cordoba Muñoz
| Challenge: | Pretrained language models are increasingly being used for many tasks. |
| Approach: | They propose to use verbalizers to help interpret masked word distributions into output predictions. |
| Outcome: | The proposed approach outperforms models trained with individual templates while using significantly less resources. |
Enhancing Hindi Feature Representation through Fusion of Dual-Script Word Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Pretrained language models often neglect the integration of different scripts within a language, constraining their ability to capture richer semantic information. |
| Approach: | They propose a dual-script enhanced feature representation method for Hindi . they combine features from Devanagari and Romanized Hindi Roberta . |
| Outcome: | The proposed method improves model performance across multiple natural language processing tasks. |
Online Infix Probability Computation for Probabilistic Finite Automata (P19-1)
Copied to clipboard
| Challenge: | Probabilistic finite automata (PFAs) are statistical language models used in natural language processing. |
| Approach: | They develop an asymptotic algorithm to compute the infix probabilities of each prefix of a string from streaming data. |
| Outcome: | The proposed algorithm improves the infix probabilities of a weighted automata from streaming data. |
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance. |
| Approach: | They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings. |
| Outcome: | The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model. |
LoRAExit: Empowering Dynamic Modulation of LLMs in Resource-limited Settings using Low-rank Adapters (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing tasks, but deployment on resource-limited settings remains a challenge. |
| Approach: | They propose a dynamic inference architecture that leverages low-rank adaptors for efficient deployment of LLMs. |
| Outcome: | The proposed architecture significantly improves performance when deployed on resource-limited settings. |
That Slepen Al the Nyght with Open Ye! Cross-era Sequence Segmentation with Switch-memory (2022.acl-long)
Copied to clipboard
| Challenge: | Grammar, vocabulary, and lexical semantic shifts take place over time, creating diachronic linguistic gap. |
| Approach: | a Chinese cross-era learning framework incorporates era-specific linguistic knowledge into a neural network. |
| Outcome: | a new learning framework for Chinese word segmentation improves on four corpora from different eras. |
Mitigating Shortcut Learning via Smart Data Augmentation based on Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to improve shortcut learning performance are limited by manual definition of shortcuts and inherent confirmation bias during model training. |
| Approach: | They propose a method of Smart Data Augmentation based on Large Language Models to identify shortcuts and generate their anti-shortcut counterparts. |
| Outcome: | The proposed method shows an improvement of 5.61% across various natural language processing tasks. |
An Effective Deployment of Contrastive Learning in Multi-label Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on contrastive learning in natural language processing tasks have not explored the effectiveness of the technology. |
| Approach: | They propose five novel contrastive losses for multi-label text classification tasks that exploit the complexity of the input logic and the semantic representation space. |
| Outcome: | The proposed contrastive losses improve multi-label text classification tasks and can be adapted for multi-task tasks. |
Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG (2025.acl-long)
Copied to clipboard
Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Zilei Wang, Weiqiang Wang, Liang Wang
| Challenge: | Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources. |
| Approach: | They propose a method that conditions large language models to generate answers even in the absence of reliable knowledge. |
| Outcome: | The proposed approach balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems. |
Evaluating Code-Switching Translation with Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown they can match or surpass finetuned models on many natural language processing tasks. |
| Approach: | They propose to use in-context learning and pivot translation to improve code-switching translation. |
| Outcome: | The proposed models show strong ability for cross-lingual understanding in a code-switching setting. |
Establishing a New State-of-the-Art for French Named Entity Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task consisting in identifying text spans that denote named entities such as person, location and organization names. |
| Approach: | They manually annotated the French TreeBank with information related to named entities . they sketch the underlying annotation guidelines and provide a few figures about the annotations . |
| Outcome: | The French TreeBank is the main source of morphosyntactic and syntactical annotations for French. |
Adaptive Contrastive Knowledge Distillation for BERT Compression (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods for BERT implicitly learn discriminative student features by mimicking the teacher features. |
| Approach: | They propose a new knowledge distillation approach called adaptive contrastive knowledge distilling for BERT compression using hidden state features in BERT as explicit supervision to learn discriminative student features. |
| Outcome: | The proposed approach improves on multiple natural language processing tasks. |
A Knowledge-Aware Sequence-to-Tree Network for Math Word Problem Solving (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for solving math word problems ignore background common-sense knowledge . a novel knowledge-aware sequence-to-tree (KA-S2T) network incorporates external knowledge and global expression information. |
| Approach: | They propose a knowledge-aware sequence-to-tree network that incorporates external knowledge and global expression information into the problem. |
| Outcome: | The proposed model can achieve better performance than previous models on a Math23K dataset. |
The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers (2023.acl-long)
Copied to clipboard
Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, Eyal Shnarch
| Challenge: | Existing approaches to apply language models to tasks that require intermediate representations are less informative. |
| Approach: | They propose a novel approach that utilizes the contrast between layers to improve text generation outputs. |
| Outcome: | The proposed approach mitigates degenerative behaviors of the model in open-ended generation, significantly improving the quality of generated texts. |
LBC: Language-Based-Classifier for Out-Of-Variable Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have great success in natural language processing tasks such as response generation, but their performance on tabular data tasks has been limited due to their inferior performance compared to traditional machine learning models (TMLs). |
| Approach: | They propose a Language-Based-Classifier (LBC) that maximizes the benefits of LLMs to outperform TMLs on OOV tasks. |
| Outcome: | The proposed model outperforms TMLs on OOV tasks by using three key methods. |
Is Language Modeling Enough? Evaluating Effective Embedding Combinations (2020.lrec-1)
Copied to clipboard
Rudolf Schneider, Tom Oberhauser, Paul Grundmann, Felix Alexander Gers, Alexander Loeser, Steffen Staab
| Challenge: | specialized embeddings are not available for tasks like entity linking or paragraph classification. |
| Approach: | They evaluate whether universal embeddings can be complemented by specialized embeddables. |
| Outcome: | The proposed embeddings outperform state-of-the-art embeddables without any fine-tuning. |
A Mixture of h - 1 Heads is Better than h Heads (2020.acl-main)
Copied to clipboard
| Challenge: | Evidence has shown that multi-head attentive neural architectures are overparameterized. |
| Approach: | They propose a multi-head attentive neural architecture that “reallocates” attention heads to different inputs. |
| Outcome: | The proposed model outperforms baselines on machine translation and language modeling tasks. |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |
Interpersonal Relationship Labels for the CALLHOME Corpus (L18-1)
Copied to clipboard
| Challenge: | a lack of corpora makes exploration of this problem intractable, says nicolaus mills . mills: communication is one of the most invaluable tools humans have . |
| Approach: | a new study uses a corpus of interpersonal relationship labels to help identify relationships . a set of labels is available for download on the website of the cnn.org team . |
| Outcome: | a new set of interpersonal relationship labels is released for the CALLHOME English corpus . the labels are available for download on the cnn.com website . |
CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing commonsense knowledge graphs are limited to English, hindering research in non-English languages. |
| Approach: | They propose a Chinese CKG generated from multilingual PLMs that is translated into Chinese . they propose 'generate-by-category' strategy to reduce invalid generation . |
| Outcome: | The proposed CKG has high quality and diversity, surpassing the direct translation version of similar English CKGs. |
A Language Modelling Approach to Quality Assessment of OCR’ed Historical Text (2022.lrec-1)
Copied to clipboard
| Challenge: | a language model-based approach is used to score the quality of OCR transcriptions in the British Library Newspapers corpus . a corpus of genre-adjacent texts captures the common and legal parlance of nineteenth-century London . |
| Approach: | They propose a language model-based approach to score the quality of OCR transcriptions in the British Library Newspapers corpus parts 1 and 2 . they aim to link newspapers of crime in nineteenth-century London to the Digital Panopticon . |
| Outcome: | The proposed approach is based on the Proceedings of the Old Bailey Online corpus, which captures the common and legal parlance of nineteenth-century London. |
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for tuning large language models from dense to MoE face significant data requirements and require large-scale post-training. |
| Approach: | They propose an upcycling instruction tuning approach for tuning a dense pre-trained model into a MoE instruction model using genetic algorithm and parameter merging. |
| Outcome: | The proposed approach improves the performance of large language models with a small amount of seed data and improves their scaling. |
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show zero-shot performance on unseen multimodal tasks . instruction tuning has yet to be explored for vision and multimodal task. |
| Approach: | They propose a multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. |
| Outcome: | The proposed model performs well on unseen multimodal tasks and is highly scalable. |
Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have demonstrated state-of-the-art predictive performance when adapted into a range of language understanding tasks. |
| Approach: | They propose to use salient information extracted a priori from training data to complement the task-specific information learned by the model during fine-tuning on a downstream task. |
| Outcome: | The proposed model can provide more faithful explanations across four different feature attribution methods compared to vanilla BERT. |
Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities when leveraging in-context learning. |
| Approach: | They propose a method that discretizes uninformative tokens using a self-supervised pre-training technique. |
| Outcome: | The proposed method achieves state-of-the-art performance across classification tasks while requiring only 0.8% decrease in performance. |
LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing (2023.findings-acl)
Copied to clipboard
| Challenge: | Deploying Transformer networks on resource-constrained edge devices is challenging. |
| Approach: | They propose a low-rank factorization initialized by SVD-based weight transfer and parameter sharing to compress and accelerate Transformer networks. |
| Outcome: | The proposed method achieves similar performance to the baseline Transformer with 3.8 times and 1.8 times fewer parameters and achieves 2.3 times speedup and 1.5 times speed up respectively. |
Algorithms for Weighted Pushdown Automata (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing dynamic programming algorithms for PDAs often resort to a PDA-to-CFG conversion. |
| Approach: | They propose to use a pushdown automaton to reduce the space requirements by a factor of |Gamma| or the runtime by reducing the number of states. |
| Outcome: | The proposed algorithms reduce the space requirements by a factor of |Gamma| or reduce the runtime by fewer states. |
A deep-learning framework to detect sarcasm targets (D19-1)
Copied to clipboard
| Challenge: | Existing methods for sarcasm target detection are difficult to implement in natural language processing. |
| Approach: | They propose a deep learning framework for sarcasm target detection in predefined sarkastic texts. |
| Outcome: | The proposed framework improves accuracy and accuracy in match and dice scores compared to the current state-of-the-art framework. |
The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (2025.findings-acl)
Copied to clipboard
| Challenge: | Prompts are essential for guiding model output and influencing content generation. |
| Approach: | They propose to attack models with prompt leakage and prompt jailbreak attacks . they summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks. |
| Outcome: | The proposed methods summarize the experimental setups and examine the relationship between prompt threats and prompt injection attacks. |
Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings that consider context have attracted great attention for natural language processing tasks in recent years. |
| Approach: | They propose two different approaches to integrate contextualized word embeddings with transformer encoders for sentence similarity modeling. |
| Outcome: | The proposed model outperforms the feature-based approach on six datasets. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
German Also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress on a wide range of natural language processing tasks, but they still suffer from hallucinating information in their output. |
| Approach: | They propose to use an annotated dataset to detect hallucinations in german news summarization and open-source it to foster further research on hallucinosity detection in german. |
| Outcome: | The proposed model can detect hallucinations in the output and evaluate the faithfulness of the summaries. |
Downstream Datasets Make Surprisingly Good Pretraining Corpora (2023.acl-long)
Copied to clipboard
| Challenge: | a dominant practice is to fine tune large pretrained transformer models using smaller downstream datasets . performance gains are not always attributable to the use of external data in massive amounts . |
| Approach: | They propose to use the same (downstream) training data for pretraining and finetuning to compare models. |
| Outcome: | The proposed model outperforms standard pretraining on the BookWiki corpus on 7 and 5 datasets. |
Explainable Text Classification with LLMs: Enhancing Performance through Dialectical Prompting and Explanation-Guided Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing explanation methods that generate keywords may be less effective due to missing critical contextual information. |
| Approach: | They propose a new method to generate explanations for possible labels using LLMs and a dialectical prompt. |
| Outcome: | The proposed method significantly improves accuracy and explanation quality over state-of-the-art methods on multiple datasets from diverse domains. |
Explicit Planning Helps Language Models in Logical Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that use pre-trained large language models to perform multi-step logical reasoning have been unable to perform this task. |
| Approach: | They propose a system that uses language models to perform multi-step logical reasoning and incorporates explicit planning into the inference procedure. |
| Outcome: | The proposed system outperforms other competing methods on multiple datasets and significantly outperformed chain-of-thought prompting on the PrOntoQA dataset. |
Contextualized Semantic Distance between Highly Overlapped Texts (2023.findings-acl)
Copied to clipboard
| Challenge: | Conventional semantic metrics are based on word representations and are vulnerable to disturbance of overlapped components with similar representations. |
| Approach: | They propose a mask-and-predict strategy to evaluate the semantic distance between the overlapped sentences using words in the longest common sequence as neighboring words and use masked language modeling to predict their positions. |
| Outcome: | The proposed method outperforms the state-of-the-art in domain adaption by a huge margin. |
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements. |
| Approach: | They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms. |
| Outcome: | The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks. |
Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Existing solutions to alleviate hallucination have considered utilizing LLMs’ inherent reasoning abilities to alleviating hallucinism, such as self-correction and diverse sampling methods. |
| Approach: | They propose a counterfactual multi-agent debate framework that predetermines LLMs' stances to override their inherent biases for answer inspection. |
| Outcome: | Extensive experiments on four datasets of three tasks demonstrate the superiority of the proposed framework over existing methods. |
G2: Guided Generation for Enhanced Output Diversity in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to enhance output diversity but compromise quality of outputs. |
| Approach: | They propose a training-free plug-and-play method that enhances output diversity while preserving generation quality. |
| Outcome: | The proposed method enhances output diversity while maintaining an optimal balance between diversity and quality. |
How Gender Interacts with Political Values: A Case Study on Czech BERT Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models are trained on large text corpora that contain value-burdened content and often capture undesirable biases, which the models reflect. |
| Approach: | They propose a method to measure the model's perceived political values by comparing Czech with a representative value survey. |
| Outcome: | The proposed method does not assign statement probability following value-driven reasoning and there is no systematic difference between feminine and masculine sentences. |
TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in natural language processing tasks but are vulnerable to harmful content and being exploited for malicious purposes. |
| Approach: | They propose a framework to measure the risk coverage of alignment datasets across three dimensions: Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |
| Outcome: | The proposed framework measures risk coverage across Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |
Improving Span Representation by Efficient Span-Level Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality span representations are limited by subset of tokens . span-span interactions should play an important role in span encoding, authors argue . |
| Approach: | They propose to introduce span-span interactions and more comprehensive span-token interactions to improve span representations. |
| Outcome: | The proposed model outperforms baseline models on span-related tasks and shows superior performance. |
iQUEST: An Iterative Question-Guided Framework for Knowledge Base Question Answering (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models suffer from factual inaccuracies in knowledge-intensive domains. |
| Approach: | They propose a question-guided KBQA framework that iteratively decomposes complex queries into simpler sub-questions and integrates a Graph Neural Network (GNN) to look ahead and incorporate 2-hop neighbor information at each reasoning step. |
| Outcome: | The proposed framework improves on four benchmark datasets and four LLMs. |
Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on various natural language processing tasks. |
| Approach: | They find that after prompt tuning, some neurons encode task-specific skills . they also show that skill neurons are most likely generated in pre-training . |
| Outcome: | The neurons are highly predictive of task labels after prompt tuning for specific tasks. |
Multi-Source Multi-Type Knowledge Exploration and Exploitation for Dialogue Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on identifying specific types of dialogue knowledge and utilizing corresponding datasets for training, but lack generalization capabilities and computational resources. |
| Approach: | They propose a framework that explores multi-source multi-type knowledge from LLMs by leveraging diverse datasets and exploits it for response generation. |
| Outcome: | The proposed framework exploits multi-source multi-type knowledge from LLMs to generate coherent, informative, and fluent responses. |
Balancing out Bias: Achieving Fairness Through Balanced Training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reducing group bias do not account for correlations between author demographics and linguistic variables, limiting their effectiveness. |
| Approach: | They extend a method for countering group bias using balanced training by balancing each demographic group in training and using protected attributes as input. |
| Outcome: | The proposed model outperforms all other methods when combined with balanced training. |
RoBERTuito: a pre-trained language model for social media text in Spanish (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models have been used in many natural language processing tasks . some domain-specific models have shown to improve performance in some domains . however, for languages other than English, such models are not widely available . |
| Approach: | They present a pre-trained language model for user-generated text in Spanish . it is based on 500 million tweets and has some cross-lingual abilities . |
| Outcome: | The model outperforms models trained on over 500 million tweets on a benchmark in spanish and english. |
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub (2025.acl-long)
Copied to clipboard
Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Cheng Qian, Zihe Wang, Yujia Qin, Yining Ye, Yaxi Lu, Chen Qian, Zhong Zhang, Yukun Yan, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing approaches lack flexibility to address diverse and ever-evolving user queries in open domains. |
| Approach: | They propose to evaluate LLMs on open-domain knowledge that requires tools to solve diverse and ever-evolving user queries. |
| Outcome: | The proposed system outperforms baselines in the open domain task-solving benchmark. |
MorphAGram, Evaluation and Framework for Unsupervised Morphological Segmentation (2020.lrec-1)
Copied to clipboard
| Challenge: | Unsupervised morphological segmentation is beneficial for many natural language processing tasks. |
| Approach: | They propose a framework for unsupervised morphological segmentation that uses Adaptor Grammars. |
| Outcome: | The proposed framework achieves state-of-the-art results across languages of different typologies, from fusional to polysynthetic and from high-resource to low-resourced. |
Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking (2024.findings-emnlp)
Copied to clipboard
Byoungjip Kim, Youngsoo Jang, Lajanugen Logeswaran, Geon-Hyeong Kim, Yu Jin Kim, Honglak Lee, Moontae Lee
| Challenge: | Existing LLMs are limited in their ability to incorporate feedback from an environment. |
| Approach: | They propose an LLM agent that consists of an Actor and a Critic. |
| Outcome: | The proposed agent outperforms existing LLMs on benchmark environments and shows that it can generate diverse trajectories and pick the most rewarding trajectory. |
ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on static and isolated representations, overlooking their dynamic evolution across layers. |
| Approach: | They propose a method which captures the cross-layer evolution of hidden states and propose 'ICR Probe' which capture the evolution of the hidden states. |
| Outcome: | The proposed method achieves superior performance with significantly fewer parameters and ablation studies offer deeper insights into the underlying mechanism of the method, improving its interpretability. |
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023.findings-acl)
Copied to clipboard
| Challenge: | Sentence-level representations are beneficial for various natural language processing tasks. |
| Approach: | They propose a generative embedding inversion attack that reconstructs input sequences based only on their sentence embeddeds. |
| Outcome: | The proposed model outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as original inputs. |
Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proficient in natural language processing tasks, but their deployment is limited by extensive parameter sizes and computational demands. |
| Approach: | They propose a method to enhance computational efficiency in large language models by 4-bit weight and 8-bit activation quantization. |
| Outcome: | The proposed techniques significantly boost task accuracies to levels comparable with full-precision models. |
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data. |
| Approach: | They propose a calibration method that incorporates synthetic data without compromising accuracy. |
| Outcome: | The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks. |
Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks, including text summarization, multi-modal machine translation, and code generation and understanding. |
| Approach: | They propose a benchmark to evaluate mLLMs’ ability to process and reason over culturally diverse procedural texts in multiple languages. |
| Outcome: | The proposed benchmarks show that mLLMs struggle with culturally contextualized procedural content, especially in low-resource languages, and perform better on multiple-choice tasks presented in conversational formats than on direct questions. |
HIPO: A Hierarchical Prompt Optimization Framework with Task Awareness and Fine-Grained Debugging (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for prompt optimization apply the same prompt across all samples . existing methods ignore variation in sample difficulty . |
| Approach: | They propose a framework that shifts the paradigm from dataset-level to sample-level optimization. |
| Outcome: | The proposed framework outperforms baselines on 27 tasks and reduces API calls, token consumption and overall cost by 1.2 to 80. |
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)
Copied to clipboard
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev
| Challenge: | In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory. |
| Approach: | They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs. |
| Outcome: | The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset. |
Outlier-weighed Layerwise Sampling for LLM Fine-tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a powerful tool for processing complex natural language processing tasks. |
| Approach: | They propose an approach to fine-tune LLMs with outliers and a gradient low-rank projection to increase the number of fine-sampled layers without a proportional increase in memory costs. |
| Outcome: | The proposed approach outperforms baseline approaches while being more memory efficient. |
A Graph-Theoretical Framework for Analyzing the Behavior of Causal Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have popularized causal language models . but their internal behavior remains poorly understood due to the high cost and reliance on large-scale benchmarks . |
| Approach: | They propose a graph-theoretical framework for analyzing causal language models . they construct graphs from model outputs and use metrics to capture linguistic features . |
| Outcome: | The proposed framework provides a macroscopic view of the overall behavior of a language model. |
MSCFFN: A New FFN with Multi-Space Cross to Accelerate Transformer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models to reduce computation complexity are limited in some areas . a new structure to reduce the computation complexity is proposed to accelerate Transformers . |
| Approach: | They propose a new feed forward network structure which splits matrix space to smaller space to reduce computation complexity. |
| Outcome: | The proposed model can achieve a faster speed and better accuracy on the long-range arena benchmark. |
Fast Quiet-STaR: Thinking Without Thought Tokens (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have achieved impressive performance across a range of tasks, but further gains require more than scaling up model sizes or training data. |
| Approach: | They propose a method that gradually reduces the number of thought tokens . this method allows models to internalize more abstract reasoning processes . |
| Outcome: | The proposed framework preserves the benefits of token-level reasoning while reducing computational cost. |
Improving Word Alignment Using Semi-Supervised Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing word alignment methods rely on labeled data, but augmenting training with pseudo-labeled data improves performance. |
| Approach: | They propose a semi-supervised framework to improve word alignment methods . they use pseudo-labeled data from multilingual encoder models as word aligners . |
| Outcome: | The proposed framework outperforms the current state-of-the-art binary alignment method on word alignment datasets. |
Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models (2025.findings-acl)
Copied to clipboard
Haoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Li Qing, Lei Chen
| Challenge: | Existing benchmarks focus on linguistic competence or structured mathematical problem-solving, neglecting fundamental numerical reasoning required in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate numerical capabilities for large language models . they use a dataset to assess number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
| Outcome: | The proposed benchmark evaluates six fundamental numerical capabilities: number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
Priority on High-Quality: Selecting Instruction Data via Consistency Verification of Noise Injection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for instruction selection rely on external models or rules, overlooking the intrinsic association between pre-trained model and instruction data. |
| Approach: | They propose a method that utilizes noise injection to identify the quality of instruction data without relying on external models. |
| Outcome: | The proposed method outperforms the model trained on the entire dataset and established baselines. |
CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Publicly available datasets can be used to evaluate performance of large language models . however, contamination of test data can artificially inflate model performance . |
| Approach: | They propose a Contamination-resilient Evaluation strategy that updates data with real-world knowledge. |
| Outcome: | The proposed evaluation strategy can be used to update datasets with real-world knowledge. |
TicTac: Time-aware Supervised Fine-tuning for Automatic Text Dating (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models that ignore the temporal relatedness of documents are time-agnostic and therefore fail to perform in automatic text dating. |
| Approach: | They propose a supervised fine-tuning model for automatic text dating that captures temporal semantic information and uses a contrastive learning-based approach to model two types of temporal relations of diachronic documents. |
| Outcome: | The proposed model outperforms state-of-the-art models on two diachronic corpora and captures temporal semantic information. |
Surprise Calibration for Better In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for in-context learning apply fixed class priors across all inputs . existing methods rely on retraining and retrain models . |
| Approach: | They propose a Bayesian-based method to capture the temporal dynamics of class priors . they identify "surprise" as an informative signal for class prior shift . |
| Outcome: | The proposed method outperforms existing methods on a range of benchmark tasks. |
Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | generative large language models (LLMs) can perform in-context learning . machine translation (MT) has been shown to benefit from in-constitu examples . |
| Approach: | They propose a compositional translation paradigm that replaces naive few-shot MT with similarity-based demonstrations. |
| Outcome: | The proposed paradigm replaces naive few-shot MT with similarity-based demonstrations. |
Uncertainty-Aware Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in a wide range of tasks, but generating factually accurate outputs remains a challenge. |
| Approach: | They propose a method that dynamically adjusts model contributions at each decoding step based on uncertainty. |
| Outcome: | The proposed method significantly improves factual accuracy and reliability over existing methods. |
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations (2025.findings-acl)
Copied to clipboard
| Challenge: | Structured matrix representations of large language models are a promising way to reduce the number of parameters in natural language processing tasks but require a significant amount of computational and memory resources. |
| Approach: | They propose to utilize invariance of the network output under certain orthogonal transformations of weight matrices to identify transformations that improve compressibility of weights within structured classes. |
| Outcome: | The proposed approach is applicable to various types of structured matrices that support efficient projection operations. |
SCOPE: Preserving Modality-Specific Cues to Mitigate Modality Laziness in Multimodal Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to learning multimodal representations emphasize shared semantics and overlook modality-specific cues. |
| Approach: | They propose a framework for learning complete multimodal representations using shared and practical cues. |
| Outcome: | SCOPE outperforms SOTA benchmarks on four datasets and achieves 27.10% accuracy improvement. |
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings (2025.acl-long)
Copied to clipboard
| Challenge: | Text embedding models are used for various natural language processing tasks such as sentiment analysis, text clustering, and content-based information retrieval. |
| Approach: | They propose a synthesis framework that leverages large language models to generate diverse negative samples with varying levels of similarity with the query. |
| Outcome: | The proposed framework achieves state-of-the-art performance surpassing existing synthesis strategies with synthetic data and when combined with public retrieval datasets. |
UzbekVerbDetection: Rule-based Detection of Verbs in Uzbek Texts (2024.lrec-main)
Copied to clipboard
| Challenge: | Verb detection is a fundamental task in natural language processing that involves identifying the action or state expressed by a verb in a sentence. |
| Approach: | They propose a rule-based approach for verb detection in Uzbek texts based on affixes/suffixed rules. |
| Outcome: | The proposed method outperforms existing methods on a dataset of Uzbek texts and has an F1 score of 0.97. |
Measuring and Mitigating Media Outlet Name Bias in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored the potential political biases of large language models, but limited attention has been devoted to the effects of media outlet names. |
| Approach: | They propose to quantify media outlet name biases in large language models and leverage this metric to develop an automated prompt optimization framework. |
| Outcome: | The proposed framework mitigates media outlet name biases, offering a scalable approach to enhancing the fairness of LLMs in news-related applications. |
From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan (2026.acl-long)
Copied to clipboard
Lei Yang, Leiyu Pan, Bojian Xiong, Renren Jin, Shaowei Zhang, Yue Chen, Ling Shi, Jiang Zhou, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuowen Han, Zhuo Chen, Yuqi Ren, Deyi Xiong
| Challenge: | Large language models have achieved remarkable success across a wide range of tasks, yet their performance remains heavily biased toward high-resource languages. |
| Approach: | They propose a pipeline for advancing Tibetan language modeling through multilingual continual pre-training with Tibetan, Chinese, and English. |
| Outcome: | The proposed model outperforms open-source and Tibetan-focused models on diverse tasks. |
Edit-Aware Reward Modeling for Chinese Grammatical Error Correction (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work has applied reinforcement learning with rule-based rewards to grammatical error correction tasks, but these methods fail to capture fine-grained quality distinctions among correction candidates. |
| Approach: | They propose an Edit-Aware Reward Model that explicitly incorporates edit-awareness into preference learning for CGEC. |
| Outcome: | The proposed model outperforms rule-based models on CGEC and other NLP tasks by 5.41 and 1.80 points. |